With a slight tweak to the first sentence, the only possible difference in interpretation becomes the referent of the person who stayed for dinner in the first sentence.
1. He would rather he hadn't stayed for dinner.
The sentence is ungrammatical without the addition of the second pronoun.
But it's possible, although unlikely barring further context, to interpret the second he as referring to somebody other than the first he.
John didn't like celebrities, and he didn't like having to deal with them staying for dinner. He certainly never got along with Elvis when he was alive. He would have rather he hadn't stayed for dinner that one night years ago.
Here, in the phrase in italics, the first he is referring to John, while the second he is referring to Elvis. It can be ambiguous, but the context of the surrounding passage makes the meaning clear.
2. He would rather not have stayed for dinner.
In this sentence, he is only used once, and the person referred to as having stayed for dinner is the narrator of the sentence. In short, there can be no ambiguity.
But assuming that the first and second he in the first sentence refer to the same person, then both sentences mean the same thing, and the only difference is one of style.