When I can use "are" or "do" to ask a question? For example if I want to call a restaurant to ask if they open or not during a holiday, which question is right?
Are you open today?
Do you open today?
They are different questions.
You would ask "are you open today?" if you are calling/emailing/messaging when you would expect them to be open (if they were open that day), or at least when you would expect staff to be in. For example, calling a shop during their normal opening hours, or a restaurant once they should be serving.
You would ask "do you open today?" if the shop had never opened before and you wanted to find out if today was their opening day - but not only then. When else you would use it depends on dialect. I've spoken to people who's native dialect would use it if you were calling, emailing etc before you would expect them to open, such as calling a restaurant in the morning before it opens for lunch. For that situation, other dialects use "will you open today?" or "will you be open today?".
In your situation, to be safest, you want "are you open today?"
Grammatically, the word open has different roles in the different versions. In the first, it is an adjective. In the second, it is a verb. In the third, it is still a verb, and in the fourth it is back to being an adjective.