TL;DR GO to refers to the journey there; BE to refers to the experience of being there, having intentionally gone there.
To be to a place is to travel there (by foot, by horse, by vehicle, etc). In the broadest sense, you take deliberate steps to get there. The place is your destination.
If you had been kidnapped, say, and taken to Boston, a city you'd never set foot in before, you would not say "I've been to Boston" except facetiously, as you did not take deliberate steps yourself to get there. You cannot say that you have experienced Boston, having intentionally gone there.
Although it is not used much nowadays, you can say:
I am to London.
which means "I am going to London" or "London is my destination".
When you want to say that you made a trip to London in the past:
I was to London on a business trip ten years ago.
When you want to say that you count London among the places you've visited, or that you recently were there:
I have been to London.
Excluding the sense "to attend" (as, for example, a school, or musical instrument lessons) to go to a place is to travel to it.
I go to London every two weeks on business.
Ten years ago I went to London every two weeks on business.
I have gone to London on business.
Brevity of visit and recency of visit (your "rule") are not essential to the meaning of go:
There used to be a family with three sets of twins living in this house. Do they still live here in town?
--Oh, no. They've long since gone to London.
P.S. With respect to summits, like the Eiffel Tower, which you ask about, or the peak of Mount Everest, we can inquire whether someone has made their way to the top:
Have you gone up?
or has been present there (has had the experience of being) up at the top:
Have you been up?
Someone who had parachuted down to the peak of Mount Everest might say:
I've been up, but I haven't gone up.