"Maybe they think they'll get invited to dinner if they're late." (said Aunt Petunia)
"Well, they most certainly won't be," said Uncle Vernon, and Harry heard him stand up and start pacing the living room. "They'll take the boy and go, there'll be no hanging around. That's if they're coming at all. Probably mistaken the day. I daresay their kind don't set much store by punctuality. ...
I'm not sure how to understand "That's if ..." in the sentence. Is it a common expression? I've looked it up, but I haven't found a reference to indicate it's a set phrase/expression. What does it convey?