We distinguish between them in much the same way that a French speaker distinguishes between masculine and feminine. They are learned. As with real grammatical gender in French, sometimes it is easy to guess whether a noun is countable or not. It is easy to guess that "femme" is a feminine word, just as it is easy to guess that "water" is (usually) uncountable. But in other cases it just has to be learned: "chip" is countable, "rice" is uncountable. "Bread" is non-count, but "loaf" is countable.
However being countable or not is not a proper gender system, because many nouns can be used as both count or non-count with a change in meaning. "A paper" means "a short report, usually scientific" (and it doesn't mean a leaf of paper in book). On the other hand, "paper" is the substance that is written on, and substances are normally non-count. Similarly "A water" is a single serving of water, whereas "water" is the substance that is drunk.
When a word has a basic meaning of "type of substance" it is non-countable. The countable form often means a specific form and not just any countable piece of that substance. So "a glass" is a cup made from glass, and not just a sheet of glass. There is no way to guess that "a glass" means a type of cup. You just have to learn it.
In the same way, you just learn that "a paper" means "a scientific report" and not any piece of paper.