The most common use of infinitives is, as you've recognized from the pattern in your examples, when they follow another verb. Most of the time, when this happens--if the verb is transitive--the infinitive is the either the direct object or it's part of a phrase that is the direct object. (e.g., I [have/had] agreed to ride with her I agreed to do what? To ride with her).
And while they're technically verbs, infinitives can function as nouns/adjective, just as gerunds can. So ask yourself, What part of speech is this?
e.g., To be prepared first is a good idea [subject is to
be prepared; direct object is a good idea]
If it's a complement, an object, or the subject, infinitives can be used.
Also, I use bare infinitives frequently in the subjunctive mood, which is another very common use.
e.g., I suggest you be home when I get there. Were I you, I'd be
It seems that I have to remember which verbs are followed by
infinitives or gerunds.
Yes, that's correct, although I wouldn't necessarily call them gerunds (because they're not used as nouns/adjectives). Words with the -ing suffix function as verbs when they immediately follow an auxiliary. They're used to form the continuous/progressive tenses.
But for other verbs like admit, allow, agree, appear and so on. It
seems that I have to remember which verbs are followed by infinitives
It's still the same rule. The infinitive functions as a different part of speech.
I am trying to agree with you. [present progressive; direct
object is to agree]
You appear to be angry. [object is to be angry]
She has been allowing him to sleep on her couch. [present perfect
progressive; object is to sleep (on her couch)]
I was planning to go out. [past progressive; to go out
They were walking to lose weight. [to lose weight is the object]
I have been drinking to get drunk. [present perfect
progressive; to get drunk is the object]
Before I began to careen down the mountainside, I had been trying to ski. [to careen down the mountainside is the object of the dependent clause; to ski is the direct object of the past perfect progressive had been trying]
Here are two examples of an infinitive as a complement:
To bodly go where no man has gone before, one has got to be prepared [subject is one; object is to be prepared; complement
is to boldly go where no man has gone before]
To eat spicy food, you've got to have a tolerance for burning sensations. [subject is you; object is to have a tolerance;
complement is to eat spicy food]
If it can be replaced with another noun or adjective, including gerunds, an infinitive can be used.
At least I know how to use "can't help" from this song " but I can't
help falling in love with you...."
You could rewrite that as I can't help but to fall in love with you. But to fall in love and falling are both the objects of [can't] help. Sometimes you need to alter the syntax, though, as I've done in the above example. As long as you're using the correct syntax--and they're not used as verbs--you can usually substitute an infinitive for a gerund.