I highly doubt that it would have a negative effect on your listening skills. Even without listening attentively to the lyrics, you will most likely improve. However, listening to songs should not be the only thing you do to improve. As mentioned in the comments on your question, songs often contain slang. They can also contain wrong stress patterns in order to fit into the music.
So, listening to music is not the only thing you should do to improve your listening skills. In addition, you could watch tv series or films. Those will contain more correct uses of English, depending on what type of character is being played.
Your best source for well-spoken English is probably the news. News anchors are trained people who are required to speak eloquently and correctly. Other than improving your listening skills, this will also provide you with knowledge of what is happening around the world.