They both can be used to ask about the basis of an observation that something is wrong. However, there is a subtle difference.
The question What's with him? almost always indicates that the speaker thinks that there is something wrong with him or his behavior.
The sentence What's up with him? can be used to mean the same thing, if there is a context indicating that there is a concern. However, this sentence can be used in a neutral, non-critical way.
I know Jim was trying to decide about his next job. What's up with him?
I can be a simple inquiry, where What's with him? is almost always negative.
Often the inflection is used as a clue. For the phrase What's with him? the emphasis is often a very exaggerated stress on the with as in
What's WITH him?
It also could be an exaggerated emphasis on him
What's with HIM?
In the neutral use of the alternative, there might be a slight emphasis on two words
What's up with him?