Short answer: It (eventually) makes sense, and it's probably grammatically correct.
Long answer: It's a terrible sentence that requires multiple readings to comprehend. As you have been told, this passive form of "go", to mean "travel", is rarely (if ever) used. Example:
We went the long way.
is standard, but:
The long way was gone by us.
is weird. The passive "gone" is easily confused with the past participle "gone", as in "missing" or "nonexistent":
All the chocolate is gone! Who took it?
When you say, "the long way was gone," it sounds like someone stole off with it. Alternately, in your example, "had to be gone," suggests the speaker believes the road is missing for some reason. If either is not your intention, you're better off using a different verb that has no ambiguity:
This was the long way, which had to be followed by us (e.g. to avoid the eponymous dangers of the shorter route though Bandit Forest)
There are many other verbs that would work as well: walked, run, traveled, navigated, journeyed, etc.
(Edit) Again, this use of "gone" is grammatical but not recommended. Except, as Lambie says, with creative writing -- but if you are fluent enough in English to write good stories, you really don't need me to tell you that it's a deliberate verbal idiosyncrasy that should be used consistently throughout the narrative.