I think the format goes like ...
....not only X, but also Y OR ...not just X, but also Y
Here, in this format, X and Y are usually grammatically parallel.
...not only X (noun), but also Y (noun) OR ...not only X (verb), but also Y (verb)
Examples -
He is not only a doctor (X), *but also a player (Y): X and Y both are nouns.
He was not just eating (X), but also crying (Y): X and Y both are verbs.
But then that's not the issue in first sentence. In the second case if 'Y' is noun, 'X' should be the noun as well and your second sentence makes the verb 'leaving' as 'X'.
He is leaving not just the institute (X), but also mathematics (Y) - X and Y both are nouns
He is not leaving (X) just the institute, but also mathematics (Y) - X is verb and Y is noun. Note that the placement of only/just is also not with not what the general format says.
Does the second sentence imply that he is not going to leave any of them, so he is staying? - No, it does not mean that.
Both the sentences mean the same (that he's leaving both - the institute and mathematics) and anyone would understand that but still, I'd prefer...
He's leaving not just the institute, but also mathematics.