According to Wikipedia:
The modal verbs of English are a small class of auxiliary verbs used mostly to express modality (properties such as possibility, obligation, etc.). They can be distinguished from other verbs by their defectiveness (they do not have participle or infinitive forms) and by the fact that they do not take the ending -(e)s in the third-person singular. (emphasis added)
The principal English modal verbs are can, could, may, might, must, shall, should, will and would
That explains the were in your sentence: Would that he were mine.
However, another place where you might see this is with the subjunctive mood. Grammarist says:
The subjunctive mood is used to explore conditional or imaginary situations.
and goes on to say:
Since statements in the subjunctive mood exist outside time, tense applies differently. The tenses of the indicative verbs could, and the subjunctive verb indicating the imagined action would not change.
That may be a bit confusing, but that explains why you might see, for example:
If only he were mine!
instead of
If only he was mine!
Merriam-Webster explains it this way:
The subjunctive is so grammatically unobtrusive as to be hard to notice: in most verbs it calls for a lack of inflection, so it's only noticeable in a context that otherwise calls for inflection. For example, the verb visit in the indicative "I visit that fabulous cat" has the same form as in the subjunctive "They suggested that I visit that fabulous cat." But if we replace I with she, the subjunctive form of the verb visit is noticeably different: in the indicative we have:
She visits that fabulous cat.
but in the subjunctive it's:
They suggested that she visit that fabulous cat.
That might explain some of the other cases where you see this from time to time.