The below was my poor attempt to point out what another answer successfully did. To paraphrase the Cambridge dictionary: The word "town" can be used without an article when discussing a town whose identity is obvious.
== Previous bad answer ==
It's a phrase based on the pioneer history of the United States and is shorthand for "the nearest populated place". When people are really spread out and travel is difficult, you don't need to say "I'm going in to Charlottesville" you can just say "I'm going in to town". Literally everyone that lives in the area knows what you mean and where you are going.
So, for the same reason you wouldn't say "I'm going in to the Charlottesville", you don't say "I'm going in to the town".
The phrase is still very much in use in the rural US.