toI was in class room and wanted to go for toilet. So without taking permission of my teacher I went to toilet. When I returned, my teacher asked me Where did you go?. And I answered.
I went to toilet.
But after that one of my friend come to me said I should have used
I had gone to toilet.
So I explained him that had should be use when we talking about past of past, but here is only one past. For this he said there are two past. i.e. 1) You went to toilet and 2) You returned to class.
So my question is which version of answer is correct here? Why?