Strictly speaking, the grammar is correct. However, it's an awkward construction because of the extraposition syntax you're trying to use, as Colin's answer explains. It's not normal in speech or writing to have such a heavily conjugated "to be" verb, immediately followed by another "to be" (here as part of a passive verb).*
Here's a better way to phrase this, and still establish emphasis through extraposition:
The more vital the worker, the more likely it is that they were asked to postpone their annual leave in the pandemic.
You can play with the tense of "they were asked" however you like — use "they have been asked", "they will have been asked", etc., depending on your exact meaning.
You also asked about dropping the "to be" in your original sentence. That would not be grammatically correct, since it is part of the passive verb "to be asked".
*The essence of the problem here is that English verb conjugations rely so heavily on helper words. When a conjugated form is used together with an infinitive — in this case a passive one — there are just too many pieces to parse without effort. In cases like that, it's often better to reword the sentence to use a subordinate clause, like I suggested.