Your instincts are correct, I'm a native English speak living in England and have never heard this word used. There's no reason that this word cannot be used, the definition fits the context in which you are using it, I just have not heard it used in common language. I found this googling, maybe it can be applied to formal speaking or theater arts.
- My interlocutor on the phone said that...
- His interlocutors all expressed an opinion that...
The definition reads "a person who takes part in a conversation or dialogue." I would therefore use
- My counterpart on the phone said
- The person I was speaking to said
- The person on the other end of the phone said
- The person I was conversing with said
- The speaker on the phone said
- His contact said via telecon
- His colleagues all expressed an opinion that
- Everyone he spoke to expressed an opinion that
- His close friends expressed an opinion that
- His confidants expressed an opinion that
- Those he conversed with/ questioned/ asked expressed an opinion that
- The conversation participants all expressed an opinion that
- His advisers expressed an opinion that
Thinking about it, in common English, the "person on the phone" or "people who are canvassed for an opinion" are often referred to specifically, in the context of the conversation. For example, if it is an informal consultation the people participating in the conversation would be "friends", if formal "colleagues". We could use "person who speaks" but even if it a stranger they would likely be classified, like "telemarketer" or "cold caller".