"Pony" is indeed often used as an umbrella term for horses, or rather more specifically, a young horse. In fact this usage is so common that many people fall under the mistaken belief that pony is just another word for a foal.
That said, it should be safe to assume, given the technical definition of pony as well as the colloquial understanding, that if someone uses the term "pony" they are likely speaking about a small horse. Of course, there are also those that use the word for any horse, in which case you'll have to rely on context for more information (or ask them directly "what kind of horse").
So, in short, your suspicion that the word is being over/innaccurately used is correct. As far as the actual line of thinking that led to this colloquial definition, I couldn't find any reliable sources explanating it, so it's anyone's guess. My theory would simply be that most people (in the US at least) are only passingly familiar with horses, and so wouldn't know that there are specific kinds horses known as ponies, and thus when hearing the word simply form the association "small horse = pony = young horse" etc.
but of course, this is just speculation