Your first example is idiomatic; your second is not.
"Late getting back" is an elision of "late in getting back," where the gerund is used as complement to the preposition. Here, "in" carries a meaning something like "related to."
I was late in realizing that he was secretly a werewolf.
"To" does not have this meaning. The preposition "to" is used with "late" only where the complement is a noun:
I was late to the party.
I was late to my own wedding.
or with the adverb "too" to indicate that the lateness was excessive:
I was too late to save her.
He arrived too late to meet the famous molecatcher.
"So" does not work in these sentences the way that "too" does, so your second example would not be idiomatic in most English dialects.