For what it is worth, I intuit this has to do with listening being a continuous activity— thus warranting a progressive tense only. For this reason the past perfect tense as well as the past continuous tense is the proper tense. Of course meanings of these two tensed versions would differ.
This is not to say listening would never take the past simple. Specifically, this means it's restricted to a point in time only. A person might say to another person I listened to you the last time, now you should listen to me.
I think the bottom line is that tenses can be tricky as there can be overlap at times.