I would favor the form without "have", specifically:
If he had known that it would upset you, he would have been more careful.
This is because, althoguh this is a hypothetical or subjunctive statement, we should be careful to keep track of the real and unreal parts, and the point-of-view in time.
that "It upset you" is real. That did happen. what is unreal is the "he knew" and the consequent that would have followed is his being more careful, and acting differently.
The (real) events that occured are:
- He did something
- That something upset you
(It may be that he felt bad afterwards, but the sentence doesn't say or imply that. He may not even know (yet) about the upset.)
The Hypothetical events are
- He realizes that a certain action would upset you
- Therefore, he is careful, and acts differently.
The point of view of the sentence is looking back to the moment before his action, where he could have realized (but did not) that it would cause you to be upset. Thus the event of the upset occurring is in the future of that moment. Therefore, I think that "would upset you" is better than "would have upset you", as "would have" is generally used when the speaker is looking back at an event which did not happen, not looking forward from such an event.
If he had known that it would have upset you, ...
suggests to me that the upset had already occurred, and that his hypothetical action would be not to prevent or avoid the upset, by acting differently and not causing it, but rather to deal differently with "you" to allow for the upset which had already occurred. Without additional context this is a slightly unlikely meaning, and so an unlikely form of words to use.