OED definitions include...
Nought or nothing reckoned as a number denoted by the figure 0
That is of the amount expressed by zero, i.e. none at all;
in Math. also transf. applied to a value of a function corresponding to the value 0 of the variable(s).
Hence (colloq.) more widely as adj. in the sense ‘no, not any’.
[emphasis mine].
That's to say, by traditional definitions and usage, OP is correct in saying that zero is a number - so it shouldn't be used as a quantifier, in contexts where you wouldn't use any other number. But in fact it's commonly used that way colloquially (usually only in spoken informal contexts).
Having said that, there are restrictions on its use...
?It's zero good hoping people will accept this as a reasonable utterance, even in casual speech.
Actually, I suppose it's possible some native speakers (certainly not me! :) will accept that contrived example. But I'm betting the vast majority wouldn't, so my advice to learners would be to avoid the form completely and always use no instead. You'll never be wrong that way.