Were is used in the Type-2 conditional sentences in the If-clauses and usually combines with the conditional present tense (would+ root verb) in the principal clauses. So, we can write: "If he were my brother, I would help him." Here, were implies an unreal hypothetical situation of "he being my brother" irrespective of any given time (which never occurred nor possible).
In the Type-3 conditional sentences, the past perfect form of the verb (had+past participle form of the verb) is used in the If-clause, while the principal clause takes the conditional perfect tense form. Thus, it is correct to write: "If he had been my brother, I would have helped him." This is also the same hypothetical situation that seemed possible in the past. This expresses a sense of regret.