You can use any of active voice or passive voice, just use them correctly.
Active voice
I want to get rid of his / their / ... talking.
It implies that ONLY the talking is annoying.
I want to get rid of him / them / ... talking.
It implies that you might want to get rid of the people also, not only the talking.
Passive voice
I want to get rid of being talked to.
(It is correct already in your question)
However, I would recommend the use of to avoid instead of to get rid of.
I want to avoid him talking.
"To get rid of" implies that the object is physical, while that talking cannot be seen / touched.
It's time to get rid of this old sweater.
or:
She finally got rid of him.
See Merriam Webster Dictionary