This depends on what city or town you're talking about. Since you mention the US in your question, I'm going to answer from that point of view.
Manhattan, a part of New York City, famously has all three. According to NYCSubwayGuide, Uptown is the northern part (anything north of 59th Street), Downtown is the southern part (anything south of 14th Street), and Midtown is the part in between (between 59th Street and 14th Street). This is from the tradition in the northern hemisphere of referring to the cardinal direction north as "up" and south as "down". Additionally,
While Uptown, Midtown, and Downtown are geographic regions of Manhattan, the words uptown and downtown can also mean your direction of travel. If you head north or towards the Bronx or Queens, you can say you are headed "uptown"; if you head south or towards Brooklyn, you can say you are headed "downtown."
However, many towns only have a "downtown", which is basically the business district of the town and generally doesn't have anything to do with the cardinal directions (traditionally, such districts are centrally located). In fact, Cambridge Dictionary's American definition of downtown specifically references the "central part of a city". This is the most common meaning of downtown in the US.
On the other hand, the town where I live now only has an "Uptown". The town is built on a hill; our central business district is called "Uptown" instead of "downtown" because it runs along the ridge of the hill, so you must literally go up to get to Uptown from almost any other part of the town. Thus, when you say "I'm headed uptown" here you may be going either north or south, unlike in Manhattan. Another example of this is Charlotte, North Carolina, which also calls its "downtown" "Uptown" for similar historical reasons (and because they think it sounds cooler).
Unfortunately, this means that the only way to know for sure what is meant by one of these terms for a particular city or town is to look it up or ask a local. However, in most cases "downtown" will mean the city's main business district, wherever that happens to be.