The most natural, informal ways to make it clear that you are not referring to the person with whom you are speaking would probably be:
- I didn't/don't mean you.
- I wasn't referring to you.
You could also use the word "implying" as you suggested, but to imply something is to say it by inference and not directly. This might be appropriate if you named somebody else but the person to whom you were speaking might think it was aimed at them, but if you have not named anyone then you should probably make it clear you were not referring to them.
Another idiomatic way you can exclude someone from a generalisation is to say:
Present company excepted.
This sounds formal, but actually is so common that it has found its way into informal speech as well.
People around here are rude. Present company excepted, of course.