Would it be possible to change "had gone" into "went"?
I think it would depend on when it was said—before or after the event.
Said sometime before:
(Two hours ago) She invited me to dinner. But it has been two years
[now] since I [only once] went to her house. So I am afraid I may lose
Told sometime after:
(Once/Yesterday) She invited me to dinner. But it was two years since
I [only once] had gone to her house. So I lost my way.
(Once/yesterday) She invited me to dinner. But it had been two years
since I [only once] went to her house. So I lost my way.*
*Note: the pattern is either "it had been… since I went" or "it was…since I had gone". Also, I don't think that the pattern "it had been…since I had gone" is incorrect even if it is far less common.
Having said that, I can't but agree with @user3169: No matter how you look at it, the distinctions are rather small and overfine to delve too deeply into.