One typical online dictionary (Dictionary.com) includes as definitions for 'take on', to accept as a challenge) and for 'take up', to accept, as an offer or challenge , so there's not a lot of difference there.
The first possible difference, as Hearth said in a comment, is that 'take on' implies that the challenge was given to you by someone, while 'take up' implies that you have accepted it (or 'taken it on' (
I would add a second possible difference, possibly related to Hearth's suggestion, and that 'take on' sounds (to me, at least) more urgent than 'take up'.
Intriguingly, Google Ngram Viewer shows that 'take/taking/taken on a challenge' are more common than 'take/taking/taken up a challenge', but 'takes/took up a challenge' are more common than 'takes/took on a challenge'. At the same time, 'take/takes/taking/took/taken up the challenge' are all more common than 'take/takes/taking/took/taken on the challenge'. So actual usage is mixed.