What I am going to write may come across as verbose and stating the obvious, for which I apologise in advance. So here we go...
Why can't I do this?
The short answer is because.
But if you are one for long answers:
Because hundreds of years ago some people started saying, "I had no trouble finding a place to stay," and this form was copied/adopted by others. Was it a calculated move by the first few people to use "finding" and not "to find"? Most probably not. Was it totally random? I don't think so. Languages - though extremely inadequate and reductive forms of communication - have their own internal pervasive but loose logic, so loose at times that we would be better off perhaps calling it "tendency" rather than logic.
Think of the subject of the development of languages and their present forms as history rather than physics or mathematics.
While learning physics, you can argue, you can ask why, you can claim that an alternative is perfectly valid. You can't do that with well-established historical facts. What happened, happened. It could have happened differently but it didn't. The present state of things is the result of what happened.
So what with your actual question?
Is there a logic there? Probably yes. Maybe not.
Is it a good idea to be looking for the logic. I don't think so.
As a non-native speaker of English I set myself some rules years ago which I think have served me well and which you may wish to consider:
- Whatever a native speaker says is right unless proven otherwise.
- Whatever I (i.e. a non-native speaker) say is wrong unless proven otherwise.
- If something "doesn't feel right", first double-check to make sure you have the right form and then keep repeating and using it until the day it "feels right".
I suppose I have not told you anything you don't already know and I apologise if I have wasted your time.