I am going out on a limb, here, so take this answer more as a suggestion than a certain, correct, answer.
There is no rule. It's one of those things that one comes up against when learning languages: you ask "what's the rule", and the teacher says: "there isn't one, you have to learn every case". This is very noticeable with prepositions.
To explain why there does not appear to be a rule, consider the case:
John makes me cry.
'makes' is not a verb of perception, and yet the above is a natural way to make that statement.
You cannot say: "John makes me to cry", or "John makes me crying".
Now replace 'makes' with another verb that fulfils exactly the same function - induces, forces, cajoles. (They do not have precisely the same meaning but their function is the same.)
You cannot say: John forces me cry / John induces me cry / John cajoles me cry.
You can say: John forces me to cry, etc.
But not : John forces me crying.
Now, take the case of 'hates'
Correct: John hates me to cry / John hates me crying
Incorrect John hates me cry.
All of these things come quite naturally to a native English speaker, so it's hard to believe that if there is a rule, no one can work it out.
Hence the assertion: There is no rule.