In the example sentence, dozen is an adjective of quantity. It modifies bananas.
Merriam-Webster agrees that dozen can be an adjective. Not sure why Oxford omits that.
An interesting point came up in the comments, so I'll use that to improve the answer.
I assume that you have no problem seeing that "ten" is an adjective. Therefore you must agree that "hundred" is also an adjective. But "hundred" has exactly the same problems as "dozen." I can say "I bought ten bananas" but I cannot say "I bought hundred bananas."
The reason is that although most bare adjectives are also complete adjective phrases, a few of them (e.g. dozen, score, hundred, thousand) are not. You need some other determiner like "a" or "the" to complete the phrase. Adjectives combine with other words to make adjective phrases. Adjective phrases can then modify nouns (helping make noun phrases) and do the other things you think of adjectives as doing. Adjectives (in modern linguistics) do not directly modify nouns; only phrases do.
All of the objections to "dozen" failing as an adjective are actually objections to it failing to be an adjective phrase. It is an adjective; it's just not a phrase.
Examples: We can coordinate "a dozen" with another adjective phrase.
"There were a dozen or more bananas."
That clearly shows that "a dozen" is not a noun phrase in this sentence because "more" is not a noun phrase. If you agree that "more" is an adjective phrase in this sentence, then "a dozen" must also be.
The distinction between words and phrases is critical for understanding the behavior of nouns and verbs, but we can usually ignore it for adjectives. This is just one of those cases where we can't.
Final note: strictly speaking, words like "ten" "dozen" and "more" are called "determiners" not "adjectives" by linguists today, but that's a complication I didn't want to introduce.