I think it depends how "opposite" you want it to be.
Immune to the charms of
I've always had a soft spot for red hair on a man, but I'm immune to the charms of an ear piercing.
It does nothing for me
I have a soft spot for a man who plays the piano, but guitar playing does nothing for me.
Those would generally be understood as you being either indifferent, or slightly negative. If you're trying to give the idea of a stronger dislike I'd say
I've never been fond of
I have a soft spot for hamsters, their little cheek pouches are very cute. I've never been fond of gerbils though — their tails scare me.