Both are absolutely correct
and in most cases, they are interchangeable. Unfortunately, like many other things in a language, it depends on the context.
If I am a college student living in a dormitory, and I ask my room mate
What are you studying?
They might tell me which subject they are studying at that moment.
I'm studying Math right now, but I'll be studying Spanish in an hour.
If I ask my room mate
What do you study?
He/she might respond with a broader answer, since I didn't use the gerund (present participle) verb tense. To say studying
suggests that the study is happening right now.
I study microbiology
In another context, let's say that you are attending a family reunion. You're going to be around a lot of relatives, and generally, people that know you. At such an event, it is inevitable that a question comes about your education, assuming you are in college/school.
Consider your uncle asking you
It is great that you're in college. What are you studying?
In this sense, the uncle is not interested in getting a course name for an answer, he's interested in the broader answer: microbiology (from above)
Also, in this sense, both questions mean the same thing. So if your uncle says, instead
It is great that you're in college. What do you study?
You would answer with the same answer.