For example, in the expression "I don't know whether I'm coming or going", would saying if in place of whether be a big mistake? Would doing it in written English be an error too?
In general, if and whether can be substituted for each other. In certain situations, they have different meanings. For example:
I don't know whether I should arrive on Friday or Saturday
I don't know if I should arrive on Friday or Saturday
The first sentence implies that the person will arrive, but they are not sure when. The second sentence could be read that the person might not arrive at all.