It is better to ask when we do use it than when we don't, as exceptions are not usually an exhaustive list.
"The" is a determiner. You use it when something needs to be determined! When something does not, you don't usually use it, unless the subject requires an article.
For example, it is quite common and idiomatic for people to say they are "going to town", or "going into town". Obviously, there are many towns - but when it is tacit that they are going to their town, or the nearest town, there is no need to determine which town it is with an article. However - if you were talking about your visit to a different area, it would be expected that you would say you visited "the town" there - because it is not the town you would normally refer to.
In your example, you refer to "The Tower of London" as "the Tower". As has already been answered, "The Tower of London" is a proper noun that includes the article. However, in abbreviated form "the Tower" needs an article because "tower" alone is not a proper noun. You could say "I visited a tower" if there were many towers in London, but there is one very famous tower and so it requires the definite article.
"Harrods" is the complete name of the famous store, so no need for an article.