The first case means
(1) Right now you wish, contrary to fact, that you knew him (subjunctive)
I wish I knew that guy. All the guys I know are either boring or stupid. (I want to get to know him now but I can't figure out how to make it happen)
or
(2) You generally wish that in the past you knew him, presumably before some other events that you are about to explain happened.
I wish I knew that guy back in college. (College has passed but I could still conceivably get to know him)
The second case has a sense of finality about the time period being discussed. You wish you had known him a particular point in the past but either it's no longer possible to know him or some other event has transpired such that knowing him now is no longer relevant. For example
I wish I had known him before he died. (too late now)
or
I wish I had known that guy before I tried starting my business. He could have given me some good advice. (too late now)