✅ "What does this mean?"
This is correct for the context you described. There are other things you could say, of course, but "What does this mean?" is perfectly fine for the situation.
🚩 "What is this meaning?"
On the other hand, while this is technically valid and grammatical English (AmE), it's meaning is actually very different than your intention. Furthermore, the circumstances you could use it in are extremely rare, to the point where it may as well be considered plainly wrong.
Your intention is that you want to parse it as "What is this" and then add "meaning" as a separate verb clause, however, in American English you cannot do this. In American English, "is" is generally followed by an adjective or a noun phrase, almost never a verb.
Instead, we parse it as "What is (this meaning)?" where "this meaning" is a single noun phrase. We are actually referring to "this meaning" as a thing, perhaps a written definition on a piece of paper for instance, or just the idea of a meaning in our heads. We are asking what it is. With the zero article, we are talking about the abstract concept of a meaning, implying that you already comprehend it.
This is obviously a very strange thing to say in almost any context, as if you have "meaning", you generally don't need to ask what it is. But that's with the zero article. If we use an article, we can clarify that the meaning in question is not understood.
🤔 "What is the meaning?"
Here, we're getting closer to your intention for the situation. "Meaning" here is still not a verb, but can be understood as a meaning that you don't understand. It's still awkward, as the definite article suggests that you are talking about a previously referred to meaning, or a meaning that you and the listener both know is the current subject of conversation. Here, we have not yet established what meaning is being referred to. So now we need to change it to:
✅ "What is the meaning of this?"
This is much clearer, and much more natural and idiomatic English. Both this and "What does this mean?" are fine.
Alternatively, in this situation you could simply use an interjection of surprise.
✅ "What?"
✅ "What on earth?"
✅ "What the (insert your choice of profanity here)?"
✅ "What is this?"