The noun "wear", meaning physical changes from use ("you can see the wear on that item"), and the adverb "worn" (something "looks worn") should be understood by anyone with a basic English vocabulary. People might say that something like a bar of soap, a sole of a shoe, a pencil, etc., "shows wear" or "will wear down" and may eventually "wear out".
However, as asked in the question, the phrase "wear somebody down" is a metaphor. It takes physical wear and applies it to human feelings. Obviously, a person's thinking or beliefs do not experience physical wear. However, a person can experience different feelings from repeated (non-physical) "pressure", in the sense defined in a comment above. This is not that advanced in terms of vocabulary. But it does require more advanced conceptual understanding. If you said to a child, your shoe is worn down, they would likely understand. If you tell a child that work has worn you down, they might not understand a lot about that statement, but they could read your emotions and facial expressions and understand basically, that work has made you tired, or left you "drained" (another metaphor).
This is a long explanation for a basic answer. In American English, from a young age we often hear phrases with "wear down"/"worn down" and generally understand what it means. I think it would be useful for non-English speakers to learn this as an idiomatic phrase, in both the physical and metaphorical senses.
