Firstly, it might be a bit confusing to speak about good memories. A good memory is simply a retentive one. Good memories may simply be clear or accurate ones. For now, let's use 'happy' instead.
Secondly, we generally speak of memories (plural): "I have happy memories of that place." Or we might say, "I never liked that place, though I do have one happy memory of it." To say, "I have a happy memory of him" is unusual, and you will certainly be asked for details: what is this one happy memory?
If your memories of X are (present tense) happy ones, it is enough to say, "I have happy memories of X."
If you tell us you had, or used to have, happy memories of somewhere or someone, we will expect you to explain why they are no longer happy. Perhaps you discovered something ghastly about the place/person.
You might also say, "I have happy memories of our first few years together: before it all went wrong."
We can talk about contrasting memories:
I had happy memories of a lake in a forest I used to visit as a child.
I went back to visit it recently and realised it was only a muddy pond in a
small wood. Now I find I have two memories (or mental pictures) of
it: as it is and as it was.
By the way, as well as happy/unhappy memories we talk about pleasant/unpleasant ones and fond ones. The expression "I remember it fondly" is useful. See the examples here.