The following quotation is from my grammar textbook under a unit about "all" and "whole":
All day/the whole day = the complete day from beginning to end, for example:
We spent all day/the whole day on the beach.
Note that we say all day (Not all the day), all week (Not all the week) etc.
But to me, "all day" in the above example sounds unnatural. I think it should be replaced with "all of the day" because it is a definite "day" not indefinite:
We spent all of the day on the beach.
All day is a fixed expression that means the same thing as the whole day. Your grammar book is correct.
All of the day is grammatical but not idiomatic in the sense of being an established expression. So your example sentence is fine. More often, people might say We spent all of that day on the beach, referring back to an occasion that they have already introduced.
Another possibility might be to say: We spent all of the day and much of the night on the beach. Here you are using day in the sense of the hours of light and not of a 24 hour period, to contrast it with night.
Finally, We spent all day on the beach says the same thing as all of the day in fewer words, and is what most people would say unless they wanted to lay special emphasis on some point.
We spent all of the day on the beach.?