"Work," in this sense, is an uncountable or mass noun, so including the article "a" in your examples is ungrammatical. "It's very challenging work" is correct in both situations.
All good dictionaries should indicate when a noun is uncountable. For example, Cambridge English Dictionary shows:
A1 [ U ]
an activity, such as a job, that a person uses physical or mental effort to do, usually for money
The [ U ] indicates the noun is uncountable.
That said, "work" also has other meanings, some of which are countable!
B2 [ C ] something created as a result of effort, especially a painting, book, or piece of music
So a musician, speaking to a friend about the difficulty of a piece of music, could say "It's a very challenging work." This is grammatical because it's referring to a specific creation rather than work in general.