At first glance the phrase "there might be a lot more of them than there were of him" might seem a bit strange, because, of course, there is only one of any individual person. However, the phrase would normally be used when there are multiple people on each side. So, for example, if Harry had been swimming with his friends Ron and Hermione, and they encountered the merpeople, you might say "there were more of them [merpeople] than there were of them [Harry and friends]. In the book here, the phrase is simply being applied to a singular noun ("him") instead of a plural noun ("them") so it sounds a bit awkward.
The sentence as a whole is actually conveying the idea that Harry had the advantage in this case, despite the fact that he was outnumbered. If you recall from earlier in the chapter, the merpeople had been preventing Harry from releasing the other hostages. They had physically overpowered him because they outnumbered him. But at this point in the narrative the situation has changed because Harry took out his wand. The sentence is telling us that because Harry was going to resort to magic, both he and the merpeople realized that Harry now had the advantage. This is due to the fact that even though Harry was vastly outnumbered, he had magic at his disposal while the merpeople did not. As is shown in the ensuing paragraphs, the merpeople are now sufficiently scared of Harry that they scatter out of his way and allow him to rescue the remaining hostage in addition to his own hostage. (Contrary to the other answer, this is saying that Harry alone had the advantage over all the merpeople combined, rather than just over an individual merperson.)