I gave him the money is simple past. You can use it to specify something that happened in the past: you can also specify when it happened, either exactly:
I gave him the money at 6pm
I gave him the money this evening
I gave him the money at about 6
I gave him the money by 6pm.
or not at all:
Yes, I gave him the money.
I had given him the money is past perfect simple. We use this when we are talking about something that had already happened before some event or action in the past. The event isn't usually a specific time: it is a part of the story, for example:
I had given him the money before I got your message
I had already given him the money, but he kept on shouting at me
Here is some more information about simple past and past perfect.
The first sentence, using simple past, is the idiomatic way of saying this. The second sentence doesn't sound right, because after you have said it I am waiting for the event that it happened before: I am not impressed by by 6pm: I want to know what happened next!
The second pair of questions present a different problem.
By 5 PM I gave him the book.
By 5 PM I had given him the book.
Starting with by 5 pm makes these sentences look odd on their own: the seem to need a clause before or after them, for example:
I was working all day, and by 5pm I had given him the book.
The deadline was 6pm, but by 5pm I had given him the book.
By 5pm I had given him the book, and then I went home.