This is an interesting question. As a native English speaker, it never occurred to me that the common phrase "let go of me" was so grammatically awkward (by modern standards, anyway). It probably reflects the speech patterns that were common at the time of its origin, which I'm guessing is way back in ye olde days. I can almost make sense of it if I close my eyes and imagine the dialog that might ensue if Shakespeare rudely grabbed someone by the wrist...
Someone: "LET GO!"
Shakespeare: "Would that I possessed the knowledge to fulfill thy command, sir. Yet verily thou hast failed to nameth yon object whereupon I am to act. Prithee tell, of what dost thou speaketh?"
Someone: "'Tis of me that I speaketh! LET GO OF ME!"
(Now that I think about it, I bet there are a whole host of linguistic conundrums that can be solved by imagining someone grappling with Shakespeare). Apologies to anyone who hasn't been exposed to the weird medieval English spoken in Shakespeare's plays-- but hey, my English teachers made me read it, so....
Anyway, back on topic. I'm not entirely sure that there are any literal differences in meaning between the two phrases in question (that is, differences that could be deduced entirely through analysis of grammar and vocabulary, without taking common usage into consideration) although it's clear that they have come to mean two different things to modern English speakers.
In my mind, the command "Let go of me" applies to a subset of the situations in which the command "Let me go" could be used-- specifically, to those situations in which the entity being given the command is physically grasping the command giver. "Let me go" would work equally well in those situations, but it would also work in almost any situation in which the command giver's freedom of movement is being restricted somehow by the command receiver.
The phrase "let go me" would definitely be considered incorrect grammar, even though it appears to contain all of the information necessary to convey the intended meaning. So, why do we need the "of"? Consider the phrase "let go, me!" That is (arguably) a grammatically-correct way of commanding oneself (or someone named Me) to let go, without identifying the object that is to be let go. The phrase "let go of me," on the other hand, identifies the object that is to be acted upon rather than the entity that is to perform the action. So, maybe that's the purpose of the word "of", then-- to indicate that we're referring to the object and not the actor. Except...
Now consider the phrase "release me!" It is correct English, even without the "of". In fact, inserting an "of" would make it incorrect. Yet, somehow the native English speaker understands that it is the object, and not the actor, being referenced. In this case, English follows the exact same grammar rules as the OP's language-- rules that in the previous paragraph were deemed incorrect.
Ultimately, I'm forced to conclude that the phrase "let go of me" is yet another "special case" in a language which, I've been told, is largely comprised of special cases. In general, the word "of" means the same thing as the word "about", so mentally substituting that word (or an equivalent word in your language) may help with comprehension. Unfortunately, I'm not aware of any general rule that you can use to determine whether the word "of" should or shouldn't be included in a particular sentence.