Your example "should have behaved" is a kind of "affected" speech that sounds formal and a bit posh or old-fashioned, like the kind of conversation you might hear spoken by wealthy characters in a period drama (either British or American, depending on location and era). It's not common in contemporary American English conversation, and I don't think I've heard it used much on British television (other than, say, programs like Downton Abbey). So just from this small piece of dialogue I imagine the characters to be people from a particular era and social class, or at least having a certain upbringing.
To answer your question: the meaning of "should have behaved" is slightly different from "behaved". "Why [someone] should have done [something]" includes a sense of proper behavior, as in what society or your peers expect of the person. For example, "I would think he should have proposed to her by now," is a formal and posh way to say, "He should propose (marriage) to her, otherwise I think it's a breach of social obligation and etiquette."
Your sentence already refers to behavior, so the only additional element is the implication of social pressure on how someone is supposed to behave. "I can't imagine any reason why he should have behaved in such an extraordinary way," is roughly equivalent to, "I can't understand why he thought it was socially acceptable to behave in that highly unusual way."
Of course we can say the same thing in contemporary (American) English, but I would use "thought it was ok" instead, "I don't know why he thought it was ok to act like that."
On a related note, Ngram says "should have behaved" was much more common around 1820 with another spike around 1880, but otherwise is appearance is pretty consistent over the past 200 years. However it's possible that many contemporary uses are from period novels in which the authors want their characters to sound more upper-class and/or British.