"Turn me off" CAN me "does not appeal to me sexually". But it can also mean "does not appeal to me" in some other way. It's quite routine to say, "Microwaving hot dogs really turns me off", "Movies that are trying to push a political view really turn me off", "People who smoke really turn me off", etc.
As perhaps you can see, sexual interest fits in with these kind of things. "Pistachio ice creams turns me off", that is, I don't like pistachio ice cream, and "Heavy eye shadow turns me off", that is, I don't like it when girls wear heavy eye shadow, are both statements of personal opinion and preference.
The phrase is normally used to express personal taste and annoyance. You wouldn't say, "Failing to consider structural load really turns me off", except as a joke, because that's a matter that's more factual than personal taste. And you wouldn't say, "Genocide really turns me off", again, except as a (dark) joke, because that's an issue that goes way beyond taste and annoyance.
Oh, and "turns off" also has a literal meaning, referring to deactivating an electrical or mechanical device. So for example a computer might display a message, "Turn me off when the backup is complete".