Umm. The part I'm having trouble with is your assertion that "him" is a possessive part. It's not. "Him" is a third-person singular pronoun used as an object.
Your tricky examples can be rephrased as
Give his book to him.
and
Give her book to her.
His and her are possessive determiners, and are followed by the referenced noun (book, in this case). Him and her are singular pronouns, and used as objects.
Your question might be better put as "Why is 'her' used both as a possessive determiner and as an object, while the masculine case uses 'his' and 'him'?"
Well, the best I can answer is "Just because". That's the way the language developed. Surely you don't think English is regular, do you?
See Wikipedia for a neat summary of pronouns.