Which is correct?:
Your question is very broad, and I doubt that it is answerable.
Your question is very broad, and I doubt if it is answerable.
Your question is very broad, and I doubt whether it is answerable.
They all seem grammatical and reasonably idiomatic American English to me. It's a matter of style. I don't think any native speaker would have a problem understanding any of them, so I'd say that they're all "correct" in that sense.
"Correct" is a difficult word to define when it comes to English usage. If it's not ungrammatical, it may be grammatically correct, but it may not be something that a native speaker normally says, so then it wouldn't be idiomatically correct. Then, too, there's technically correct (the expected answers on internationally standardized English tests like the TOEFL and IELTS), and what your English teacher says is correct on his or her tests. I've seen myriad English tests in Taiwan and Japan, and most contain some multiple-choice questions that have no correct answers or many correct answers
I'd say and write #1, I wouldn't say or write #2, and if I were to use whether, I'd have to change "doubt" to "wonder". But, as I said above, it's strictly a matter of style in this case.
You've brought up an interesting problem. The word doubt, when used as a verb, generally expresses uncertainty, along with a lack of assurance or confidence, as in:
I doubt we will win the game.
Both doubt and don't know are ways to express uncertainty, but doubt connotes less confidence, while don't know suggests an inability to form a conclusion or hunch. In other words, if the weatherman says one of these two sentences:
I doubt it will rain tomorrow.
I don't know if it will rain tomorrow.
then neither is conclusive, but I'm more inclined to leave my umbrella home in the first case, but to at least bring it in my car in the second.
So, going back to your example sentences, I'm not sure I would use any of them. If I had real doubt that the question was answerable, I'd be more inclined to say:
Your question is very broad; I doubt it is answerable.
without putting any word between "doubt" and "it."
However, if I changed the verb to don't know, to express a higher level of uncertainty, I might use one of your options then:
Your question is very broad; I don't know if it's answerable.
Your question is very broad; I don't know whether it's answerable.
All three are grammatical, but as a speaker of British English, I would normally say the second. However, doubt that occurs much more frequently in the Corpus of Contemporary American English, the British National Corpus and in citations in the Oxford English Dictionary than either doubt if or doubt whether, but we need to aim off to some extent for the occasions on which that will be a demonstrative pronoun rather than a subordinator.
The choice between if and whether in general can sometimes involve ambiguities, but these are normally resolved in speech by intonation. In writing, however, the choice can make a difference to the meaning. If someone sends us an invitation which includes the words ‘Please let us know if you’re coming’, we may think that we need to indicate our intention only if we plan to accept the invitation. If we don’t, we might conclude, there is no need to say anything. The writer, however, may have intended to us to reply in either case, and the use of whether rather than if would have avoided that difficulty.
This is an interesting question. The first point to note is there's no absolute rule of grammar involved here. OP's sentence is perfectly valid with that, if, whether, or indeed, nothing at all between the verb doubt and the statement which is doubted. The second point to note is that usage has changed...
...it seems "no conjunction at all" is making a comeback from its C18 heyday. Which accords with my own perception, and – dare I say it – seems like the sensible choice. Why give yourself the problem of choosing between the other three possibilities, if you don't need to use any of them?
In case it's not obvious, the third point to make is that none of the variations convey any difference in meaning (not even a subtle nuance, so far as I'm concerned). Nor does formal/informal come into it.
My version would be something like this:
Your question is very broad, and I'm not sure if/whether it is answerable.
The word whether doesn't apply here because of a pattern which is consistent wh- words.
We cannot doubt a wh- proposition. These are all wrong:
These are right:
In a nutshell the reason is that, Wh- words are embedded questions. For example, by itself, who will be there is a question: Who will be there?. And you cannot doubt a question! You can only doubt a statement (such as the answer to a question). A question is itself an expression of doubt.
Also, note how the presence or absence of the word not does not make a difference in I do not believe. However, the not makes a difference in: I'm not sure. This is a semantic matter rather than purely syntactic: it has to do with meaning rather than grammar.
Now about if. The word if, in this situation, is a substitute for whether that many modern English speakers use. It can also substitute for that, but these two usages are not on the same level. I wonder if it will rain (if substituting for whether) is broadly acceptable and can be used in formal writing, but I doubt if it is true (if substituting for that) is still very colloquial.
The likely reason why if instead of that is less acceptable is that if instead of whether serves a useful purpose: it provides economy of expression. if is only one syllable and phonetically easier than whether. Not to mention that it is two letters instead of seven, in writing. But the word that can be economized by a complete omission: if that is troublesome to say or write, you can usually just drop it entirely! So there is no point in replacing it with if. Quite simply, I doubt it is true.