You could, if you wanted, say "the large family of which the house...", although this would be very formal, almost stilted.
However, you absolutely can't say "the large family that its house" (at least not with the meaning you've suggested - see note below.) "Which its" and "who its" would be wrong too.
The usual and correct phrasing is "the large family whose house...".
NB. In English, if you say "the large family which its..." or "the large family that its..." or "the large family which it..." then the "it" or "its" should refer back to a noun other than the large family. For example, "A small family used to live there. The large family that its house was bought by now enjoyed the benefits" = the large family that the small family's house was bought by. (It would be more straightforward to say "the large family that bought its house", but my point is that "...that its..." can work but has a different meaning from the one you proposed.)