That sounds great to me. Very florid and poetic, if that’s the effect you’re going for. The tense is fine; standard English doesn’t distinguish the imperfect aspect the way some other languages do. “I used to sit” might make it slightly less ambiguous that the narrator is talking about a recurring or habitual former action, and “over a hundred evenings” slightly more explicit that we’re talking about many different evenings rather than one evening that felt like a hundred, but I think the meaning is perfectly clear as it is.
One minor nit: “where I sat a hundred evenings ...” is a non-restrictive clause. There isn’t some other Blue Bird cake shop the audience might have in mind where he didn’t sit a hundred evenings with his grandfather. So, in formal written English, there should be a comma before the relative clause: “The Blue Bird cake shop, where I sat a hundred evenings with my grandfather.”
The one reason I can think of that you might prefer a different construction is if if you want the narrator to sound a little less formal. Personally, though, I think it works well to say this in a way that stands out a bit. My advice would be not to change this turn of phrase into something more humdrum.