You can use the preposition despite in the structure of despite + -ing verb, but you cannot use it in front of the to-infinitive.

So the the first phrase (despite being married) is grammatically correct and the second one is not correct.

You can use the preposition despite in the structure of despite + -ing verb, but you cannot use it in front of the to-infinitive. The OP has used the word married as an adjective in the first phrase in the structure of despite + being + adjective which is also correct.

So the the first phrase (despite being married) is grammatically correct and the second one is not correct.