Notice that it is grammatical (and perhaps more idiomatic) to say "We'd recommend you book your flight early."

Thus, you have an option to use a bare infinitive in both cases. There is no general rule saying that passive infinitives must be bare although in this specific case I agree that you cannot idiomatically just plunk a "to" in front of the passive infinitive without other changes to the sentence.