In an answer to a related question, I said The guiding principle should be don't use Past Perfect unless you really have to. But OP's example here is one of those cases where you should use it.
NGrams charting Past Perfect had forgotten against Simple Past forgot tell us nothing except that the latter is more common overall, which is no more relevant than noting that Simple Present forget is more common than Simple Past forgot. Unsurprisingly, since most of us live in the present, not the past.
But note these counts from Google Books for something a bit closer to OP's specific case...
realised I had forgotten (2640)
realised I forgot (184)
(figures for noticed are similarly skewed, but with lower values)
This overwhelming preference is because realised implies an action/mental state that occurred at some specific point in time [in the past]. And since "the thing realised" was an act of forgetting, clearly that must refer to an even earlier time (as soon as you realise it, you're no longer forgetting it).
Consider these two constructions...
1: When they told him he had incurable cancer, John realised he had smoked too many cigarettes.
2: When they told him he had incurable cancer, John realised he smoked too many cigarettes.
...where Past Perfect #1 places the excessive smoking before the realisation. Perhaps John gave up some time before the fateful diagnosis, or perhaps he still smokes and will continue to do so for the limited time he now has left. The choice of tense doesn't really imply anything about that.
But with Simple Past #2, the excessive smoking was still going on when he realised this. Grammatically, there's a strong implication that he will quit the habit, even though semantically we might think that's a bit pointless since it won't cure his cancer.
As pointed out, in OP's exact construction, it's logically impossible for the "forgetting" to continue into the time-frame governed by the "noticing". But as my second pair of examples show, grammatically, that's the implication of using Simple Past. To avoid this "temporal disjunction", OP needs to use Past Perfect here.