Both are grammatical. There may be a slight preference for the gerund in this case because the reference is to an actual past event rather a future event, but I must admit my ear would not be offended at all by “to leave” or “leave” in this example. I do not agree with the comment suggesting that only the bare infinitive is strictly proper.
In general, both to-infinitives and present participles can be used as nouns.
I like to swim
I like swimming
In some cases, using the bare infinitive as a noun is grossly incorrect.
I like swim
is utterly unidiomatic.
One important exception to the general rule that both infinitives and gerunds can be used as nouns is that gerunds can be used as the object of a preposition whereas infinitives cannot.
He is very good at skiing
He is very good at to ski
He is very good at ski
are not acceptable.
In many cases, gerunds are used for completed actions, and infinitives are used for intended or contemplated actions, but this is not an invariable rule.
His leaving was wise.
It was wise of him to leave.
They are equally acceptable.
In short, except as objects of a preposition, both to-infinitives and gerunds are grammatical, but consider preferring a gerund for a completed act and an infinitive for an intended act.