"I have a bad memory" sounds a lot like I remember something bad that happened to me.
While that is grammatically speaking a correct possible interpretation, a listener will be perfectly able to understand you given the context of the conversation you're having.
Generally speaking, people are not going to state that they have had a negative past experience without either (a) the conversation clearly leading to this topic and/or (b) you not elaborating on that specific experience. If you just state that "you have a bad memory", it is clear that you mean that you don't remember things well.
If you'd still rather avoid the ambiguity anyway, you could opt for alternatives such as "my memory is bad", which more clearly refers to your memory ability (rather than a particular memory) being bad.
Alternatively, you could change the adjective to a less ambiguous one. Poor, flawed, short, ... are more clearly descriptive of your ability to remember rather than a particular memory you have.