In the first example, to "swing" is to move while or as if suspended from some central axis. Like a playground swing, where you move back and forth while hanging from a bar. "Off" here is an adverb meaning "away from" or "in a manner resulting in no longer being atop". So Harry is sitting on a broom and holding onto it, and he then moves in a swinging motion that results in him no longer being on the broom. I picture him holding on to the broom with one hand and moving in a semi-circular arc.
The second example is less clear. Is this an isolated sentence used as an example in a dictionary? It's very odd wording. We can talk of "swinging off" something in a literal, physical sense like in the first example. We often use the word "swing" metaphorically to describe someone going between two ideas, like "Jack swung back and forth between loving Sally and hating her". I would read that sentence to mean that the "stupid man" had taken the side of the enemy in some conflict, but now he has swung back to "our" side. But I've never heard someone use the phrasing "swing off a side" in the sense of a side in a conflict. You could say he "swung off the side of the building", but "swung off the enemy side", well, I think I understand the metaphor, but it just sounds awkward.