Your version of the answer is correct, because when you were answering the teacher's question, you both were in the present. So only
I went to the toilet.
If you were relating this whole story later to somebody, then you might use the Past Perfect:
When I returned, the teacher asked me where I had been. I replied that I had gone to the toilet.
The Past Perfect is possible because you are retelling this story now. The moment when the teacher asked you is in the past, and the moment then you visited the toilet is in a deeper past.
Grammar terminology: my second example uses "backshifting" of tenses. More specifically, this is called "a backshift in reported speech" (here's another helpful explanationanother helpful explanation).