These molecules can rotate in the three-dimensional space.
My feeling is that the article should be omitted. I'm not completely sure, though. What grammatical rule should we appeal to?
You are correct. Normally the word the should be omitted.
There is no general rule. By custom, we treat Euclidean three-dimensional space as if it were an uncapitalized proper noun that doesn't take a definite article, like the names of most cities: London, New York, Edinburgh, etc.
These molecules can rotate in London.
These molecules can rotate in three-dimensional space.
Other mathematical concepts work like the sky, e.g. the Klein four-group, the Petersen graph, the A6 group. There is only one of each of these, and the customary the makes us regard it as "always in view", always available to be pointed out by the (even at first mention). But A6 by itself, not as a qualifier before "group", works like the names of cities: it doesn't take an article.
However, you would need the before three-dimensional space in order to narrow down or distinguish the concept from the usual infinite Euclidean space. For example, if you had just defined several vector spaces of varying dimensions, containing polynomials, exactly one of which was three-dimensional, then you could refer to it by "the three-dimensional space". But then you would be talking about math, not molecules.
You are correct in omitting 'the' in this case. The case that keeping 'the' in the sentence would be appropriate is when describing a subset of three-dimensional space, like such:
"These molecules can rotate in the three-dimensional space inside a container."