For what it is worth, I intuit this has to do with *listening* being a continuous activity— thus warranting a _progressive_ tense. For this reason the past perfect tense as well as the past continuous tense is the proper tense. Of course meanings of these two tensed versions differ. 

This is not to say *listening* would never take the past simple. Specifically, this means it's restricted to a point in time only. A person might say to another person _I listened to you the last time, now you should listen to me._

I think the bottom line is that tenses can be tricky as there can be overlap at times.