Finding is much more idiomatic there than to find.
I believe that this is because the omitted headword is problem. Problem normally takes an -ing clause as its complement, not a to clause.
We had a problem finding anybody.
is much more common than
?We had a problem to find anybody.
The iWeb corpus has 3977 instances of problem finding, and only 521 of problem to find; but actually the disparity is greater than that, because in many of the latter, to find is not actually the complement of problem, for example "we don't need to finish the long division problem to find the remainder".
As usual with such questions, there is no rule or logic to this: it's just the way that the word problem happens to work.