When negative phrases are fronted (moved to the front of a clause), we typically apply subject-auxiliary inversion (the subject and auxiliary verb switch places).
In your example the negative phrase is "very, very few times", meaning "almost no". The word few in this phrase is a special type of negative word, in technical terms called an approximate negator (because "few" is "close to zero").
Take a look at these example from The Cambridge Grammar of the English Language (CGEL), page 95:
1a. He found not one of them useful.
1b. Not one of them did he find useful.
The first example has the negative phrase not one of them in its basic position inside the clause. In the second example, we've moved it to the front of the sentence. Since it's negative, we apply subject-auxiliary inversion. (And since we didn't have an auxiliary in 1a, we've added the dummy auxiliary do to make inversion possible.)
2a. Somewhere he mentions my book.
2b. Nowhere does he mention my book.
In this pair of examples, we can see somewhere moved to the front without inversion. This is fine, as somewhere is positive. But nowhere is negative, so in 2b we've used inversion.
So no, you should not change the book's sentence. It's just fine the way it is.