That's not the meaning of left in this sentence, though I can understand your confusion. Left here is not a form of leave; rather it is equivalent to remaining (all that is left). That would be definition #7 from your link:
leave 7. be left [transitive] to remain to be used, sold, etc
Is there any coffee left?
How many tickets do you have left?
So in this case, they're discussing Harry's family. Some people might have a lot of living family members, but Harry does not. He used to have parents, but he doesn't any longer. So if you take the family he was born with and subtract the family members who died, you're left with the family he has left or the family (members) that are remaning: the Dursleys.