Comparatives and superlatives ("better", "soonest") are adjectives that would not normally take the definite article "the". However, it is grammatical to use the adjective as a noun, where the adjective stands as an ellipsis of a noun phrase. For example:
Which of these shirts do you want?
I'll take the blue (shirt).
How many are in your group?
We are three (people).
When forced to choose between evils, try to pick the lesser (evil).
Among the runners, she is the best (of all the runners).
As in your example, the adjective can also stand for an adverbial phrase that would otherwise include a noun:
We like this book the best (of all the books).
There are also set idiomatic phrases like "the X the better", but again these are elliptical phrases that omit redundant information.
When should I arrive?
The sooner (moment in time that you can arrive), (it will be) the better (situation).
I don't think there is any special rule around the use of this other than the sentence should still make sense with the noun omitted. For example:
When ordering wine at a restaurant you should always pick the least.
The least what? It's not clear from context if I mean "the least expensive wine" or "the least fruity wine" or something else. Of course people say things like this all the time, which can require the listener to ask follow-up questions.
There are other cases where the definite article can be used before a comparative or superlative adjective, for example if the adjective has been converted into an ordinary or proper noun, as in someone's name or title. For example, the boxer Muhammad Ali was nicknamed "The Greatest".