People often don't speak grammatically correctly (or standard English, he talks with a west country style).
This is one of those times. Hagrid is a big friendly not very educated person [1], this phrase confirms that.
Your analysis of the meaning is correct.
It's they that should be sorry!
I would actually say: They are the ones who should be sorry!
[1] he was thrown out of Hogwarts early.