Of the two suggestions, the one that makes most sense grammatically is:
What colour clothes do you prefer to wear?
This makes perfect sense and would be understood. That isn't to say it is particularly idiomatic, as I don't believe there isn't really any single, common way of asking this.
People do ask:
What is your favourite colour?
This is a very broad question though, not necessarily to do with clothing. Children are more likely to ask this question, as the idea of having one single favourite colour for everything (clothes, home decor, flowers etc) is not held by most adults.
In fact, the reason why the question you are trying to ask is not all that common is probably that few people wear just one colour from head to toe.
It is idiomatic to refer to a single colour as "my colour", meaning that it is a colour which is complementary to one's looks, skin-tone etc. This would have to be used in context to make sense, for example:
That jacket really suits you. Green is definitely your colour.
So, in context (ie you are already speaking about clothes) you could ask:
What is your colour?