The most important thing to know about verbs
The role of the verb in a sentence is to indicate what the sentence claims or denies about its subject. For example, in "The cat sat on the mat", the subject is the cat, and the sentence makes a claim about the cat: that in the past, the cat sat on the mat. The words "sat on the mat" form the predicate of the sentence: what is being claimed or denied about the subject.
A verb has person, number, and tense in order to serve the role of predication. The verb's person and number "agree" with the subject in order to indicate what the sentence is making a claim about. In English, verb person and number are very simple, but in other languages (including Russian), they're more complex and their role is more obvious. The verb's tense indicates the time that the sentence makes a claim about. Because "sat" is in the past tense, "The cat sat on the mat" makes a claim about what the cat did in the past; it claims nothing about what the cat is doing now or will do in the future.
The most important thing to know about infinitives
The main function of the infinitive is to let you use a verb without predicating anything—that is, separately from making a claim. For example, in "The cat wishes to sit on the mat", the verb is "wishes", and the sentence claims that the cat is wishing something. "Sit" is in its infinitive form because it doesn't claim or deny that the cat is sitting. It only describes the cat's wish, which may or may not be fulfilled.
This is why an infinitive doesn't have person, number, or tense. The word "infinitive" comes from Latin roots meaning "unfinished". An infinitive verb doesn't perform the full role of a verb; it doesn't make a claim about a subject. It just helps describe a claim which some other verb in the sentence makes. The infinitive provides a way to use a verb separately from claiming or denying that the verb's meaning is true of a subject.
Now you can probably guess what perfect infinitives do. They describe a situation where a verb's action was complete, without using that verb to claim something about the subject.
I'm sorry to have disturbed you.
In this sentence, the main verb is "am" (which is contracted with "I"). The subject is "I". The sentence claims that I am sorry. What am I sorry about? The fact that I disturbed you. You could also say "I'm sorry I disturbed you", which uses a subordinate clause to claim that I disturbed you. In "I'm sorry to have disturbed you", the completed action of disturbing you is wrapped up in an infinitive. The fact that I disturbed you is left to implication.
An infinitive can describe a situation that exists or really did happen. It doesn't have to describe a situation that doesn't exist or didn't happen. The infinitive just lets another verb do the predicating.
One reason people say "I'm sorry to have disturbed you" is to emphasize that the disturbance has already happened and can't be undone. That phrasing contrasts with "I'm sorry to disturb you", which doesn't suggest that the disturbance is complete. You might say either sentence after opening someone's office door while they're in the middle of work, before you ask them for some assistance. "I'm sorry to disturb you" emphasizes that the disturbance is still in progress, and you have more to say. "I'm sorry to have disturbed you" emphasizes that the disturbance can't be undone (and you might have more to say, or perhaps opening the door was a mistake, and you'll close it and back out now).
Common uses are not rules
It's natural to try to simplify and organize what you're learning about a new language. Memorizing rules can be a tempting way to simplify what you're learning. But most "rules" are wrong, especially "rules" that are really just common uses. It is not a rule that "the perfect infinitive can only be used for plans that didn't happen." That is just one of many uses of the perfect infinitive. It can also be used for plans that did happen: "I am glad to have started work last week, as planned." The perfect aspect emphasizes that I am glad that my start of work is now in the past, behind me, complete.
Infinitives have many, many uses in English. They can hook up with other words in a sentence in many different ways, far more than I can explain in one message. What they all have in common is that the infinitive lets another verb make the sentence's claim. The infinitive just fits into that claim in some way.
It's not even a rule that if you want to use a verb without using it to designate the predicate of the sentence, you must use an infinitive. English also uses gerunds for that (as in "I'm sorry for disturbing you"). When to use a gerund and when to use an infinitive is one of the most confusing parts of English grammar, because it doesn't follow any clear rules.