Please, allow me to give you good reason for some of your doubts.
"Let me come in," he said to me.
He requested that I allow him to come in.
The subject of this imperative let is the implicit second person. In this example, the active voice of the direct speech is retained in the indirect reference. The change from implicit second-person to explicit first-person reflects nothing more than the change in which person speaks. His words are spoken from his perspective, and mine from mine.
"Let Ritwik come in," he said to them.
He requested that you be allowed to come in.
In this example, we see a change in voice between the original and the representation. The direct object "Ritwik" in the active voice becomes the subject in the passive voice. His original subject doesn't even appear in my report of his request.
He said, "Let us go there."
He suggested that we go there.
The voice of his statement and my report are both active. In this example, the thing that changes is which verb is primary. The verb to let isn't represented at all -- not even by a synonym like to allow -- in the subordinate clause of this report. Instead, we have a finite form of to go taking as its subject the object of let that is modified in his original by the bare infinitive phrase object complement.
You seem to be confusing and conflating these three separate effects: change of person speaking, change in grammatical voice, and change to primary verb. You won't find one rule that covers all three cases. Instead, you need to determine which and how many of these separate effects happen to apply to any given sentence.
One rule doesn't handle them all.