My question is that which of the followings is correct?
- My question is [...]
- My question is that [...]
for example should I use that in the first sentence of this post or not?
The conjunction that is never used in this context. You have a number of ways to express this question, but none of these ways employs the conjunction that.
You use the conjunction that with an indicative clause, not a question.
You may however use the demonstrative ('pointing') pronoun that with a question:
Though both can be used to ask a question, there's something interesting.
When you use that, the sentence does not require comma in general and I think question mark is optional (I'll edit this if someone comments on it, to make this answer better). On the other hand, the sentence without that generally requires comma.
Check these -
My question is, how does it go there?
My question is that how does it go there (?).
Both are correct. I think to use that in this case question of style. There are cases when using that is not valid. This case is not one of them.