You're mixing up two separate problems here.
First, you ask about "to + gerund". In this situation, "to" is followed by a noun form, not an infinitive verb, so "to" must be a preposition rather than the infinitive marker. In "contribute to" something, "to" is a preposition as well. So your first problem is how to know when to use a noun after "to", and when to use an infinitive verb. It's usually easy enough to figure out from the context whether a noun or verb comes next. If a noun comes next, use gerund, otherwise the infinitive.
The second problem you seem to be asking about is how to determine whether after a particular verb you should use a gerund (note: it's not "to + gerund", just "gerund") or "to + infinitive" (I'm inferring that "to + verb" means "to + infinitive"). This is a harder problem, as both forms represent verbal meaning.
Mostly students memorize several lists of verbs: one list of verbs that take gerund after, and second list of verbs that take infinitive after, a third list of verbs that can take either and it doesn't change the meaning, and a fourth list of verbs that can take either, but it changes the meaning. This does eventually work, but it's slow and painful because it seems there's no reason.
But I noticed a pattern I haven't seen in any book which has helped my students guess which type of verb to use. In any sentence like this, there's two verbs. Let's call them verb 1 and verb 2. So, in the example sentences: "I enjoy drawing" and "I want to visit my sister", "enjoy" and "want" are verb 1 and "draw" and "visit" are verb 2. Our job is to figure out what form verb 2 should be.
The general rule is: if verb 2 happens after verb 1, then verb 2 will usually be "to + infinitive". This means if verb 2 happens before verb 1, at the same time, or just generally, then verb 2 will usually be a gerund.
Some examples:
"I want to visit my sister."
"Visit" happens after "want", so we use "to". The same works for other verbs of the future, like "hope", "wish", "plan", "learn", "agree", "choose", "promise" etc.
"I quit smoking."
"Smoking" only happens before you "quit", so we use a gerund. Some other verbs with a past meaning are "admit", "justify", "excuse", "ban", "acknowledge", "celebrate".
"I enjoy drawing."
"Enjoy" and "draw" happen at the same time, or you might say it's generally true without any real time relationship. Some similar verbs include "tolerate", "imagine", "dislike", "end up", "can't help", and "avoid".
A big advantage of this is that for all the verbs that have different meanings depending on which form you choose, you can correctly predict the meaning if you remember that "to" means future:
"I remembered to lock the door."
This means I locked the door after I remembered, so it uses "to" + infinitive.
"I remember locking the door."
This means first I locked the door, then later I remembered, so it uses a gerund.
"I'm trying to put on more muscle."
This means now I'm trying, the future goal is to put on more muscle, so we use "to" + infinitive.
"I've tried eating more protein."
Here, "trying" and "eating" happen at the same time, so we use gerund.
On the whole, it's about 90% accurate, which I think is way better than nothing, but there are exceptions:
"Advise", "consider", "propose", "support", and "suggest" all have a future meanings, but use a gerund. "Appear", "fail", and "seem" have non-future meanings, but use "to + infinitive".
