These phrases have overlapping meanings and different nuance.
Both pay up and pay off can mean to clear a debt make a payment that is due. However, pay off is transitive, with either the debt or the creditor as object. On the other hand, pay up can be intransitive, and often is. When it is transitive, it's usually the amount of money that is the object, and you may then also get a prepositional phrase indicating who they are paying.
Pay up is also more likely to be used impolitely, or as part of an instruction or even threat.
Examples:
It's going to take me years to pay off my student loans.
I need to pay off that loan shark if I don't want my legs broken.
I can't believe there's still £2000 to pay off on my credit card.
If you don't pay up, I'm going to break your legs.
Okay, we've eaten our meal, I guess we'd better pay up and leave.
My cars going to get repossessed if I don't pay up.
Pay off also has an additional sense, also used in the sense of paying someone, but not paying back a loan or settling credit. It's used for bribes and other corrupt payments, hush money, that sort of thing.
If you want to drive through that part of the country, you've got to be ready to pay off the guards at the checkpoints.
There was a woman with a story that would have sunk the campaign, but we paid her off. She'll keep quiet for now.