Only one is correct, and that would be
What is this game called?
Since does is a "helping" verb to an other existing verb in the sentence, there is just no place for it in here. For example,
What do you call this game?
There needs to be some sort of "helping" verb in order for you to construct said question. Omitting that, you would not be able to form a grammatically correct question.
Additionally, the basic difference between what and how is that using what you are asking on the goal and using how is asking on the journey to said goal. Put it simply,
What did you achieve?
Would be question on specific final result, contrasting which
How did you achieve it?
Is asking on the "steps" that led to mentioned result.