Technically, you can listen to a sound, but you cannot really be a listener to sound. You have to be a listener of sound.
Listen is a verb describing an action, so it can be targeted or aimed toward an object. To is the preposition that expresses that.
Listener is a noun describing an instance of a type of person - "one who listens" (this is what the -er suffix means). So it can be the object of a verb or preposition, but it cannot really directly have an object.
Also, using to X in the manner like listener to sound means according to X and X can be a concrete person or abstract noun.
I am a good person to Jane = I am a good person according to Jane.
I am an enemy to uncleanliness = I am an enemy according to uncleanliness.
I am a listener to your podcast = I am a listener according to your podcast.
But since the example is "listener" and "podcast" - and "podcast" is obviously something someone listens to, it's likely to be something that is not noticed or misunderstood.