When used as a noun, a dress is normally used to refer a typical type of (perhaps fancy) clothing that women sometimes wear:
The definition from the Merriam Webster dictionary:
a piece of clothing for a woman or a girl that has a top part that covers the upper body and a skirt that hangs down to cover the legs
However, if you say to a man:
Nice dress!
...then it will most likely be interpreted as a sarcastic comment if the man is. Unless he is actually and intentionally wearing a dress, of course. However, people who do that usually want to look feminin (this is called crossdressing, a man trying to imitate the outer appearance of a woman).
If the man you're talking to is wearing a 'costume' like you said, I would just use the word costume instead. This word is used for clothing worn by someone who is trying to look like someone/something else or a type of clothing worn by a specific group of people, especially in past times.
To compliment a man on his attire, there are plenty of synonyms, search for synonyms of clothing in the dictionary and you'll find other suitable terms.
In any case, try to avoid using the word dress when talking about a man's outfit.
You could also try to be more specific, if it is a particular part of the clothing that you like:
or use a hypernym:
- Nice outfit!
- You look well-dressed!