Yes, you are using the past subjunctive and yes, you should say "If he were you".
With moods, just like tenses, we know the conjugation. That's because every conjugation happens for some combination of tense and mood. There is a never a verb which has a tense but no mood or a mood but no tense. If we don't name a mood, such as when we talk about the "past tense" it's because the most common mood, the indicative, is assumed. As in, you are indicating something, in other words saying that something exists or is true. That's got to be 95% of the things we say every day.
Don't worry too much about moods as a category -- other than conditional, they are simple and consistent in English. Studying the conditional as its own unique category should be enough because English has a great deal of nuance in conditionals. (e.g. "If you had been", "If you were to be", "If you were to have been", ...)
(EDIT: Based on some of the other answers, I have decided to clarify with further examples and discussion.)
This is not to say that "If I was" is never correct! Rather, it has a different meaning. This page explains it well. Here are two examples that I hope make it clear:
"If I were sick, I would not have come to the party." (subj.)
This is subjunctive because it tells me that the speaker does not believe he was sick and is imagining some different reality and how things would be different.
"If I was sick, I didn't know it at the time." (ind.)
This is indicative because it tells me that the speaker doesn't know whether he was sick. He is speaking (with uncertainty) about this reality in which we live, not an imagined reality where something is different.
VERY IMPORTANT: You will find a lot of incorrect usage of "was"/"were" on the internet, on TV, and in conversation with native English speakers because this is one of the most commonly ignored rules in modern English. I didn't say "most common mistakes" because it is easy to understand the meaning even if the wrong word is used. For example:
"If I was ..., I would have ..." (common, but not technically correct)
In this sentence, we know that the speaker is speaking in subjunctive mood even though he used "was", because the "would have" is unmistakably subjunctive. You should avoid this usage on a resume or in an academic paper, and probably even in important business meetings. In other situations it is generally consider informal, rather than a mistake.
In 200 years, it is very likely that "if I were" is going to be gone from the language and will be only found in historical usage. There was a time when people in England went to jail for referring to the King as "thou" instead of "you". Then for probably a hundred years, some people ignored the difference and some others people said "those fools have terrible grammar and no respect." I'm sorry for all the confusion, but this is part of any living, changing language.