Colloquially, you can be a fan (or not a fan) of anything, abstract or concrete, tangible or intangible.
When it's used colloquially, be a fan of implies "like".
I'm a fan of Tom Cruise. ~ The speaker is a fan of the actor. They surely like the actor.
I'm not a fan of spicy food. ~ The speaker doesn't like spicy food much.
Here is a good example showing that the expression "a fan of" can be used with virtually anything. It's from the movie Edge of Tomorrow (the quote is from www.moviequotesandmore.com):
[as they drive along the main road in France]
Cage: You don’t talk much.
Rita: Not a fan.
Cage: Of talking?
Rita: Not a fan of talking, no.
It just means that Rita doesn't want to talk (with Cage) in that scene.
So, your I am not a fan of negativity simply means: I don't like negativity.