1. I hate them calling her fat.
2. I hate their calling her fat.
The verb - hate - licenses Gerund-Participle form of verb. It can also license a Noun Phrase.
3. I hate him.
4. I hate Meryl.
5. I hate copying answers in the exam.
A Gerund-Participle can have its own subject. A Noun Phrase can sit before the Gerund-Participle as its subject. The Noun Phrase can be in genitive form or in plain form.
6. I remember [his/him reading my mail].
In your quoted sentence calling her fat is a Gerund-Participle. So naturally the subject can be in genitive form or in plain form. So both of your sentences are grammatically correct and mean the same thing [you hate calling her fat. Who is calling her fat? Them]. But note that even if your sentence #2 is grammatically correct, it's not very natural. And it's not often that people will say/write something like this sentence.
You can analyze sentence #1 in a different way as well. You can think of the verb - hate - taking two complements - first, them and second, calling her fat.