Given the phrase:
Please, stop talking. I asked you a question.
Should one then add:
Answer me, please OR Answer to me, please.
Generally you say...
Answer me, please
...if you are asking someone and only you both are in picture.
On the other hand, I can think of an example of
Answer to me
Suppose you found one of your employees culprit. The scene includes the employee, boss, a sweeper and you. If you are asking something and if he's answering a sweeper, you can draw his attention by telling him, Answer to me (not him). However, you put an accent on the pronoun 'me' there (Thanks Araucaria).
Answer to [someone] also works when the context is 'reporting' to someone. It simply means, that someone is the boss as DavidRicherby gives his input.
John answers to Jane says that John reports to Jane as she's the boss.
Also, you can say "my answer to him was no" but then, it's an all different situation.
I believe it used mostly as "answer to" when the recipient is a person. Ex. Answer to him/ Do not answer to them/ And just "answer" when there is no person involved; ie. Answer the door/Do not answer that letter, etc.