There are four aspects in English: Simple, Progressive, Perfect, and Perfect Progressive. Here's the wikipedia article on it.
I went to Peru.
This is the simple past. From the Wikipedia article:
Simple constructions normally denote a single action, a repeated action, or a relatively permanent state.
The simple is used to note a single, basic action. It can also be used to describe something that occurs every X units of time.
I was going to Peru.
This is the past progressive. From the Wikipedia article:
The progressive or continuous aspect is used to denote a temporary action or state that began at a previous time and continues into the present time (or other time of reference).
The progressive is used when something is started and is still going on relative to the context. For example, "When I was going to Peru, a bird hit the window of the airplane." That means that the bird hit the window while I was still going to Peru.
I had gone to Peru.
This is the perfect. From the Wikipedia article:
The perfect aspect is used to denote the circumstance of an action's being complete at a certain time.
This is used to give context to an event by describing something that happened further back in the past. For example, "I had gone to Peru before, but the second time I went to Peru I was still amazed by the culture."
I had been going to Peru.
This is the perfect progressive. From the Wikipedia article:
The perfect and progressive aspects can be combined, usually in referring to the completed portion of a continuing action or temporary state.
A combination of the perfect and the progressive. For example, "By the time I became fully fluent in Spanish, I had been going to Peru every summer for six years."
"was gone" is using the perfect passive participle as an adjective, and the sentence "I was gone to Peru" is not grammatically correct. (Not anymore at least, but that's an excursion for another day.)
Out of those four options, "*No, but I did go to Peru, which is right next door." is the most grammatically correct in this context. It uses the simple past because it is describing a single action which happened in the past, and gives no chronological context to another phrase.