Both are grammatically correct but don't mean the same thing.
The most common usage is that in your second example. Compare with this similar example found in the
We use whether … or to introduce a clause giving two options or alternatives:
"I can’t decide whether to paint the wall green or blue."
"She didn’t know whether he was laughing or crying."
In your first example, the only potential problem does not lie with your use of "whether", but with the semantics of the clause "I watch ABC or NBC". It is grammatically correct, but it does not mean the same as your second sentence. Compare with the following example:
"I don't know whether he watches ABC or NBC"
Meaning that you don't know what he usually watches.
The way you have phrased it is analogous. If you want to convey that meaning (you don't know what channel you usually watch). it would be correct.
On the other hand, you cannot use that to convey the same meaning as in "I don't know whether to watch...". Suitable alternatives for that would be using a modal verb, as you suggested, or using a future tense: "I don't know whether I will watch ABC or NBC"