Have you ever been to Japan?
Yes, I went there 5 years ago.
Is it correct to use the past simple to answer, as it refers to something happened in the past and finished? Or should I answer "Yes, I have been there 5 years ago"? Is there any difference?