Can anyone shed some light on the following aspect?

By 5 PM I gave him the book.

By 5 PM I had given him the book.

In both the sentences the book was given before 5 PM and no later than 5 PM. But I can't understand then what the difference between them is. a Also is it right to use simple past with 'by' at all? If yes, then why and if not, then why is it incorrect to use simple past with 'by'?