Spelling English words is challenging work.
Work is a non-count noun here. It refers abstractly to work in general, not concretely to individual, countable instances of work.
The noun phrase challenging work is indefinite, but you only use the indefinite article a(n) with count nouns. This noun phrase doesn't need an article.
So the sentence is fine the way it is.
*Spelling English words is a challenging work.
This would be fine if work were a count noun in this context, but it's not.
This sentence is not fine, so I've marked it with a *.
Spelling English words is a challenging task.
In contrast, task works fine as a count noun in this context. It refers concretely to individual, countable units of work.
This sentence is fine.
Spelling English words is challenging.
Here, you've replaced the noun phrase challenging work with the adjective phrase challenging. Adjective phrases don't need determiners, so of course there's no article.
This sentence is fine.