Is not too fond of a polite way of saying dislike a little or neither like nor dislike (indifferent)? Or is it used towards something that one likes a little, but is irritated to find others liking it too much?
This can completely depend on the situation.
Generally it would be used lightly to suggest you have a lack of liking without any significant dislike of something. Or that you just dislike it a little.
Eg I'm not too fond of getting the bus: I'll do it if it's the easiest option or I have to, but I'll use other modes of transport if possible
But with certain intonation, it can be far stronger and heavily laced with sarcasm. Unfortunately this requires an understanding of the context and tone, and like all sarcasm can be very dry.
Eg a fairly obvious one
I thought you hated peach cider? Yeah I'm not too fond of it
The person could either slightly dislike peach cider or completely detest it
And a more subtle one
Me: what do you think of Sandra you: I'm not too fond of her
Chances are that this person dislikes Sandra, but it's impossible to be sure if the strength.
Generally, unless it's being given with obvious sarcasm or doesn't fit with the context or the previous sentence, it's genuine and used to suggest a mild dislike, or negative preference.
Another thing to note is that typically people would only use this sort of sarcasm with friends who are ready know their opinion - eg if you know I hate Sandra or Peach Cider you may jokingly ask my opinion, in which case 'I'm not too fond' would be humorous understatement.