I was born in
As that stands, something is obviously missing. The preposition "in" lacks an object.*
the house I was born in
Here, we can find the thing that was obviously missing before, even though it's in an unusual place. The object of "in" is "the house". I was born in that house.
I was born
As that stands, nothing is missing. There is no function word that lacks an argument. There is no preposition or verb in search of some other word to make it make sense.
the house I was born
Here, there is nothing to connect "I was born" to "the house". The two sets of words have no relationship, and the reader has no idea why all those words appear together in that order.
We can describe the first case in terms of an optional relative pronoun:
the house [that/which] I was born in
And so grammar books describe a rule stating that a relative pronoun which is not the subject of its subordinate clause can be removed. However, there is no such rule about relative adverbs.
The reason is that a "cancelled" relative pronoun leaves a grammatical gap, and that gap itself shows us how the subordinate clause is related to the preceding phrase. When you attempt to cancel an adverb, there is no grammatical gap left behind. You haven't elided an argument; you simply haven't supplied an adjunct. There is nothing to attach the clause to its intended modificand.
Don't attempt to cancel a relative adverb. It leads to nonsense.
_______________
* Not every "in" requires an object. There are, for example, several phrasal verbs that use an intransitive "in": done in (meaning defeated or exhausted) and taken in (meaning deceived) leap to mind. The verb "born" doesn't have such a phrasal use. When "in" modifies "born", we naturally expect it to have a clear object.