I want to know which sentence is better of the two?
- I am not in mood to do debate.
- I am not in a mood to debate.
There are a couple of problems with your first option. Let's go over them in some detail.
While debate can be both a noun and a verb, as a noun it is not something one would usually say can be done. If you want to use it as a noun and apply a verb that indicates a debate happening, you would use to have. In that context, debate is a countable noun, so it needs an article. That, you can talking about "having a debate", but not "doing debate"
The noun mood can be used in a manner that is countable, or it can be used in a manner that is uncountable. When you talk about someone's mood using the preposition in, it is countable. Thus, you cannot be "in mood", but you can be "in a mood" or, in certain situations "in the mood". The definite article suggests that there is one specific mood (or quality of mood) in question.
So, we can modify that sentence to be grammatical, and we get
"I'm not in the mood to have a debate."
(Explaining why it's the mood, rather than a mood, is difficult; I suspect it's just one of those things you need to learn. However, in this case it wouldn't seem especially weird to use a - it's just not how a native speaker of the dialects I'm familiar with would do it.)
Your second option is much more nearly correct. That uses debate as a verb, qualifying mood and creating the noun phrase "mood to debate", meaning "mood suitable for debating". Again, however, that should take a definite article, giving us:
"I'm not in the mood to debate."
This has exactly the same structure as the corrected version of your first option, it is just using debate as a verb, rather than have.
You can also use for, followed by a noun (including a mass noun), with mood, rather than a to-infinitive. In that case, we can use debate as a mass noun referring to the practice of debating, rather than a specific instance or event. That doesn't need an article:
"I'm not in the mood for debate."
Or you can still use it with an article, indicating that it is used in its countable sense - meaning debate as a specific instance or event:
"I'm not in the mood for a debate."
Or, where it's clear what the debate would be about, it can even have the definite article:
"I'm not in the mood for the debate."
I hope that helps.
2 is better. 1 is missing important words.