After a joyous and restful three days, I left this beautiful city.
The phrase a joyous and restful three days can be changed to a joyous and restful period of three days. Now we're talking about a period, not a three days. Let's change joyous and restful to just nice.
After a nice three days, I left this beautiful city.
Dropping the a makes it ungrammatical. You can reword it as:
After three nice days, I left this beautiful city.
However, the meaning could be slightly different. Maybe I was there for a week but only three of the individual days were nice. In the original sentence using the adjectives before the number makes it clear that the three days are a group and so the indefinite article is required. That being said, you would most likely find this formation in writing rather than speech.
If you mention the nice days in an earlier sentence you could say:
I spent three nice days here. After the nice three days, I left this beautiful city.
I spent three nice days here. After the three nice days, I left this beautiful city.
As for why a is mandatory, it's the same as I saw a cat. It's ungrammatical to drop the a in that sentence too.
You can say
Three fat boys ate
A fat three boys ate
because it doesn't make sense to call the group fat, only the individuals.
Three noisy boys ate
A noisy three boys ate
In the first sentence each of the boys is noisy. In the second sentence the group as a whole is noisy but not necessarily every boy. It's possible for an individual boy within the group to not be noisy.
So the construction you asked about only works if each adjective can be applied to a group.