There are always some people who are exceptions, but yes, native English speakers in general do clearly and easily distinguish these sounds.
I'm not a linguist, but from what I've read and seen it tends to be fairly common that native speakers of a language will easily distinguish phonetic differences that affect meaning, while ignoring those that don't. I'm guessing your native language doesn't distinguish these sounds: perhaps one of them isn't used, the same letter (or equivalent) can represent either sound, or which sound is used in a given word depends on the speaker's dialect.
Most English speakers would have a similar problem learning a language in which the sounds of k as in skip (not aspirated) and k as in kill (aspirated in most dialects, almost pronounced khill) are distinguished (affecting meaning). As a native English speaker I can hear the difference if I think about it, but as far as understanding spoken English goes they're both the sound of k. (Thanks to Peter OlsenPeter Olsen for the example.)