If by “formally correct” you mean “acceptable in formal discourse” — Yes, it’s an ordinary metaphorical extension of down employed as a preposition, and according to the Oxford English Dictionary it’s been perfectly acceptable for over a century and a quarter.
FOR ADVANCED STUDENTS:
derives from an Old English expression meaning “from the hill” and had as its initial sense, according to the OED (the emphases are mine):
I. Of motion or direction in space.
1. In a descending direction ; from above, or towards that which is below ; from a higher to a lower place or position ; to the ground.
It is applied to any degree of descent, from a vertical slope to the slightest slope as in a nearly level river valley, and thus passes into sense 2, in which the descent may be entirely imaginary or conventional.
2. To some place which is conventionally viewed as lower in position ; in the direction of a current, or with the wind ; from the capital to the distant parts of a country ; away from a university ; from the House of Lords to the House of Commons or ‘lower house’ ; to a lower or inferior court of law, etc.
That sense 2 is already documented in 1200. And it’s just the second of 22 senses the original OED (1897) distinguished in the bare adverb, before it got to fixed phrases and to prepositional uses. (There are more senses in the Supplements, and I’d be willing to bet still more in the current online version.)
One of the prepositional definitions and an accompanying citation are relevant here:
2. Often with no implication of actual descent : To (or at) what is regarded as the lowest part of ; along the course or extent of. …
… 1878 Mod. Traffic passing up and down the line.
You must not ask English prepositions and adverbs of position or direction to behave strictly literally or logically—they have far too much work to do.