What is the meaning of below sentence ?
a) I asked if he and my father had been twins.
Does it mean
b) I asked if he and my father were twins.
If Yes , then when to use a) and when b) ?
If he and your father "had been" twins, then they are no longer, and the only way that can be is if either is, or both are, dead. The past progressive tense is used about an action or situation that was completed at some point in the past. It seems an unlikely situation - you have to ask if your father has (or had) a twin?
I think that the topic is ambiguous. Suppose that "he" were twins with George, and "my father" were twins with Tom. You could answer "they both were twins." But the sentence doesn't have additional context asking if "he" and "my father" were the context of "twins".
Suppose that "he" and "my father" were part of triplets, and that brother is now dead. The person being asked should answer "no" to the twins discussion, right?
You can still use the "were" construction even if one (or indeed both) are dead. Family relationships aren't somehow cancelled.
It sounds clumsy and odd to say "King George VI had been George V's son", you'd just say that he was.
"Had been" implies something else afterwards, like "My uncle had been a Regimental Sergeant-Major for ten years before he was a florist".