She [my mother] quadrupled her already fabulous record and she was awarded the fairest prize of all - a ride in an army airplane.
Oh, we were proud kids! Even vicariously this was an eminence we could hardly stand. But my poor mother - I must tell you that there are certain things in the existence of which my mother did not believe, against any possible evidence to the contrary. One was a bad Hamilton and another was the airplane. The fact that she had seen them didn’t make her believe in them one bit more.
In the light of what she did I have tried to imagine how she felt. Her soul must have crawled with horror, for how can you fly in something that does not exist? As a punishment the ride would have been cruel and unusual, but it was a prize, a gift, an honor, and an eminence. She must have looked into our eyes and seen the shining idolatry there and understood that she was trapped. Not to have gone would have let her family down. She was surrounded, and there was no honorable way out save death. Once she had decided to go up in the nonexistent thing she seemed to have had no idea whatever that she would survive it. (John Steinbeck, East of Eden)
I can see the subjunctive mood, but why does the infinitive have perfect tense? Being the subject in the sentence, it seems to have simple tense – not to go.