So, the first thing we should recognize is the author is singing about someone who doesn't/didn't love them - as you pointed out, "wouldn't have" is used when discussing something different from what actually is/was true.
There are two main ways beyond that this sentence could be interpreted in this song:
The bad things that happened to me wouldn't have mattered to me if you loved me
This version implies that if they had the subject's love, the author wouldn't have cared about other bad things they're singing about.
The things that I did that made you leave me wouldn't have mattered to you if you loved me
This version implies that the subject is rejecting the author really because they don't love him, not because of the things he did.
I looked up the lyrics, and assuming this is the song by Todd Rundgren (there are several songs with the title you gave), I think the second answer is more likely, but the lyrics are ambiguous.
There's also a third interpretation that doesn't fit well in the context of the song, which is
Nothing would be different if you loved me.
This meaning is common for uses 'it wouldn't have made a difference' in general, but very unlikely for 'it wouldn't have made a difference if you loved me', because almost always people who talk, write, or sing about love believe it does make a very large difference.