Since you are asking this in ELL, I assume you are looking for some rules, not just usage patterns of native speakers.
There are rules for forming conditional sentences known as "the three conditionals" (or sometimes four). You can find a god explanation of these rules here. They are usually taught in ESL classes as strict rules, although as you can see in this discussion, some native speakers claim they are useless and don't reflect natural use of language; and definitely some native speakers never heard of these "rules".
There is merit in rules: If you follow them, you are practically assured to be well-understood and considered correct in formal writing or speech. Assuming that's what you are looking for, let's examine your sentences...
To clarify the difference, I'll change the sentences from interrogative to plain conditional declaratives (taking the tense from the second part):
- If he hadn't tried to flirt with you yet, I'd be surprised.
- If he hasn't tried to flirt with you yet, I'd be surprised.
The first sentence is exactly in the third conditional form, which discusses unreal past. It implies that (the speaker believes) "he has tried to flirt with you" but is thinking of "what if he hasn't?". So I'd say the corresponding sentence is perfectly fine:
Has he tried to flirt with you yet? I'd be surprised if he hadn't. (OK)
The second sentence does not fit any of the conditional forms - it uses present perfect instead of past perfect - so it may be considered ungrammatical... however, the original sentence uses present perfect (correctly) in its first part (the question), it may be more natural to keep the tense when referring to the same idea in its second part. I think it would be more common in practice...
Has he tried to flirt with you yet? I'd be surprised if he hasn't. (Also good)
In summary, both will be understood with the same meaning and you are unlikely to be "corrected" either way.