Neither of these is entirely correct, but this is a matter of the adverbial with ago.
Ago always measures the timespan it designates from the present moment, 'Speech Time'. Consequently it can only be used with a present-tense verb: a simple present or a present progressive. (But ago cannot be used with the present perfect, because ago explicitly locates the eventuality it modifies in the past, and that sort of reference is prohibited with the present perfect.)
To express a timespan extending back from a past reference time we use before instead of ago:
I wore the dress I had bought three years before.
This may not work in your second example, however, since we don't know when the tree was planted.
??The tree was planted by the† people who had founded the city over 400 years before.
This means that they planted the tree 400 years before they founded the city, which seems unlikely. If you are want to say that they planted the tree after founding the city, you will have to say so expressly; the verb construction won't take care of it for you. In fact, I can't find a way to get all these ideas into one sentence without repeating one of them; this is the best I can come up with:
The people who founded the city over 400 years ago planted the tree shortly after the foundation.
† The people, because the relative clause is restrictive and defines a specific group of people.