The word late, doesn't mean "till a later time." More accurately, it means "after the desired time."
"The plane's depature is late" is no guarantee it will leave at a later time. It just means "The plane's departure is after the desired time."
Using your examples
I worked past the desired time at work today.
He stayed up past the desired time last night.
Using this understanding, you'll see that "early" means before the desired time, which makes sense.
One doesn't have to even know what the desired time is, to be late or early.
He is a late bloomer
Is an old saying, meaning, "He took longer (past the desired time) to achieve his potential." As far as I know, there's no general agreement on how long "the desired time" to achieve one's potential should be.