Grammar is like water, in that speakers will take the path of least resistance. Things begin to become ungrammatical, or at the very least unidiomatic, when the water starts flowing upstream, as it does in your examples.
I suspect most native speakers would use must there, to avoid would have had to have been:
For him to have become a threat in the teacher's eyes, he must have
been exhibiting severely bad behavior.
That's not possible in the second example. My AmE ear tells me to avoid "would have had to be doing" and to say something like this instead:
If I hadn't finished my work at school, I would have had to continue
working on it at home, while...."
or simply, would have had to finish it at home, as the semantics of the sentence don't demand the continuous.
Maybe something like this:
If I hadn't reached him before he left the office to discuss that
emergency, I would have had to be phoning him repeatedly until he
answered the call on his mobile phone.
But even then, many native speakers would find would have had to phone him repeatedly sufficient. As long as there's some indication of repetition, such as "repeatedly" or "again and again", it isn't necessary to use the continuous tense. Absent the adverbs, then would have had to be phoning him until... is more viable.