It's all allusion and metaphor, and very, very well done. The fandango is a couples dance. There is a competitive element to some versions. Or, in other words, they were figuratively speaking, dancing round each other with a bit of drunken flirting. It's about the journey to having sex.
Turning cartwheels across the floor is a figurative way of saying that you're having a hell of a happy time. Or in this case, they both are - laughing, smiling, dancing round each other.
It's a loud party. At least one of them is drunk. After some dancing around each other they wind up in the sack, maybe as a one night stand, maybe as a drunken fuck upstairs at the party. It may be her first time, hence the reference to vestal virgins. He may have wondered if her reluctance was prick teasing, hence the bit in the third verse:
"and forced her to agree saying,
'You must be the mermaid, who took Neptune for a ride.'
She smiled at me so sadly
That my anger straightaway died."
There are however regrets in the fourth verse:
"If music be the food of love,
Then laughter is its queen.
And likewise if behind is front,
Then dirt in truth is clean."
In short, though there was loud music and laughter, this was not the way to go about it. He thinks they went about it the wrong way round.
And did he later, drunkenly, brag about it?
"And so it was that later
As the miller told his tale
That her face, at first just ghostly,
Turned a whiter shade of pale."
Faces go white with fear or anger. Chaucer's Miller is drunk, and his tale concerns a poor student relentlessly pursuing and finally cuckolding his landlord, a carpenter. The story winds up with it becoming becoming common knowledge that the carpenter's wife has slept with the student, and the townsfolk mock the carpenter. So was the woman in the song already attached to someone? And is the cat now out of the bag?