The other answers do an excellent job explaining too adjective to infinitive, so I won't address that. I'll try to explain a different aspect of this, which might be what you're finding so surprising: specifically, how it could possibly be that removing an adverb could render a sentence ungrammatical? As you said, removing "slowly" from "He slowly walked down the street" changes the meaning a little but it doesn't make it ungrammatical. An adverb is just an optional modifier, so what could be wrong with leaving it out?
It's a phrasal thing
The reason removing an adverb can cause ungrammaticality is that in English, often a phrase is the unit of meaning, not the individual words. Sometimes, the meaning of a phrase doesn't derive from combining the meanings of the individual words like when you combine slowly with walked to mean slow walking. The phrase too adjective to infinitive is actually an indivisible unit of meaning. The word to doesn't express a relationship between one part of the sentence and the object of a prepositional phrase the way prepositions usually do. Instead, too and to together supply the distinctive signature of this phrase. If either of those exact words is missing, the phrase is gone. The phrase is itself like a word in the language. It just happens to consist of two words plus two slots to be filled in by other words (the adjective and the infinitive).
This drives people learning English as a foreign language crazy. In most languages, the individual words combine their meanings according to standard rules for combining meanings. English mostly does that, too, but English also makes heavy use of phrasal verbs and similar phrasal templates that are themselves indivisible units of meaning. (Actually, I think nearly all languages do this at least a tiny bit, just not to the extent that English does.)
Sometimes up does not mean up
I once spent most of a plane flight talking with a woman from Brazil. When the plane landed, I asked her, "Do you have someone to pick you up from the airport?" She said, "Why is everything always 'up' in English? Why don't you just say 'pick me from the airport'?"
It's because pick is actually a completely different verb than pick up. To pick someone means to choose them, like when children "pick" who they want on their team before playing a game. To pick someone up from the airport means to give them a ride.
Prepositions often serve as the distinctive word in the signature of phrasal verbs. Here are some more phrasal verbs that include pick:
pick on someone = bully someone, or single someone out for unfair, harsh treatment
pick up after someone = clean up someone else's mess
pick at something = repeatedly scratch or poke something with a sharp object; for example, "pick at a scab"
pick a person up = meet a person you've never met before (not a planned meeting) and go on an impromptu romantic date
pick a call up = answer a telephone call
In phrasal verbs, one of the words is always a preposition. But it doesn't function like a preposition. The up in pick up doesn't work like up in walk up the stairs, where up introduces a prepositional phrase that modifies walk. In pick up, the word up changes pick into a completely different verb! Some linguists call up a particle rather than a preposition when it plays this role, since it doesn't introduce a prepositional phrase.
It's got to be too
Here is a consequence of all this that might be surprising: you can't replace too with a synonym. "I'm excessively tired to drive" is also ungrammatical! The problem isn't that an adverb is missing from I'm tired to drive, it's that specifically the word too is missing.
It's just like pick up: you can't replace up with a synonym or approximation like above or upward without radically changing the meaning. Pick you above from the airport is actually ungrammatical. There's no phrasal verb pick above.
When I hear I'm tired to drive, I feel disoriented. The word tired seems to be leading somewhere, but then I can't recognize a familiar phrase that starts with tired, like tired of gerund. I also can't find a way to join to into a familiar phrase with anything earlier in the sentence. Is it like in order to (for the purpose of)? No, because nobody would get tired in order to drive. Is it like toward? No, because to drive is not a prepositional phrase. It's a to-infinitive. The word to doesn't mean a direction here.
The sentence is ungrammatical because the parts don't join together. No familiar phrase starting with tired concludes in to infinitive. And no familiar kind of phrase ending in to infinitive starts with tired and gets a reasonable meaning. Normal, one-word-at-a-time combining of meanings doesn't work, either.