It's literal. If psychology.stackexchange.com existed, they could help better than me, but basically, if Harry opens his mouth words may come out that indicate his true response to Dudley's appearance.
These words would very likely not include handsome, grown up and pride inspiring.
Given the context of how the mentioned protagonists already feel about Harry, he decides it is safest not to say anything at all. He may start off trying to say words that the other members of the household want to hear, but during any ensuing conversation, he will have to say something reflexively, and this might be closer to what he actually thinks. The expected response from the household to that would not be likely to be pleasant for Harry, so he literally is not trusting himself to speak.
Additionally, he has concerns about breaking into uncontrollable laughter. Devoting effort to talking would make it more difficult to prevent this from happening. I would have picked this as the reason, but in that case a better lead in sentence would be 'Harry didn't trust himself to open his mouth.', so I feel the speaking is more related to the fact that Vernon and Petunia have both uttered highly positive sentences and Harry can't carry that off. Contrast this with either of the twin Weasely brothers, for instance, who would have no problems in trusting themselves to speak.