The question posits only two possible formations of this phrase:
- The questions whose answers I don't know
- The questions that I don't know their answers
Of these, the second is ungrammatical. The first is acceptable, although many English speakers intuitively feel that "something is wrong" about using whose to refer to things instead of beings (in spite of the fact that Shakespeare himself did so in ~1600CE!)
This usage can be avoided by (as noted in another answer) using the pronoun which instead...
- The questions to which I do not know the answers
This may be preferable in formal writing. However, in idiomatic English, the most common form is...
- The questions [that] I don't know the answers to