Short answer: Native usage on this point (at least in the US) is quite a bit looser than what prescriptive grammar books teach you.
In this particular case, the emphasis is on the fact that (hypothetically) you've seen your friend, that such an action has occurred. But, because the proximity is important, the completed task is not "seeing your friend" but "seeing your friend yesterday."
The specific point in time is secondary; we don't really care that it was at 9:00 or 17:00 or whenever yesterday. And, if you can think about it figuratively, it doesn't matter that it was specifically yesterday versus last Friday or a week ago or whenever. Yesterday here isn't to designate a specific time, just to imply proximity, almost like a synonym for recently.
This usage largely results from the fact that the example is hypothetical--it's the hypothetical nature that leads the speaker to not think of yesterday as a specific point in time. The indicative version, You've seen your friend yesterday sounds fairly unnatural, albeit not totally ungrammatical. It places the emphasis on the accomplishment of having seen your friend, and again, mentions yesterday almost as an afterthought.
That's the best explanation I can give as a native speaker--the given example doesn't sound odd to me at all. I know blaming prescriptive vs. descriptive grammar is a not very satisfying answer, but it's my best attempt to explain the psychology going on in this choice of tense.