I can confirm the usage of "call on" as a way to describe a personal visit by native speakers, albeit in rather formal circumstances (a foreign dignitary discussing which meetings to hold).
The reason people have not understood you might be both because this is a somewhat old-fashioned meaning (depending on their age and circumstances, they might not have come across this phrase and will do so later in life) and because there might have been something misleading or ambiguous about your context—other things you've said.
In casual conversations like the one I imagine you've been having, a more appropriate, casual phrase would have been, "They will drop by our place tomorrow".
In general, I'd advise you not to try to use unfamiliar meanings—ones you've not encountered in real life—on the basis of finding it in a dictionary. The dictionary will give only rudimentary indications of connotations and appropriate context, if at all. Always start with a real text—either a real-life conversation, or a book, or an article, by native speakers. Then you have a context by which to judge the semantics of the situations where this particular word (or, rather, lexeme) is used.