As you say, "similar in their style" is used idiomatically, but not very often. This ngram shows that the phrase "similar in style" is used far more often. Idioms are widely accepted in everyday speech although may not be strictly "correct".
The two could be used interchangeably in some situations. However, as "their" is a possessive pronoun, it may not always be appropriate to use it if you are speaking about something as opposed to someone.
Personally, I agree with your example:
Those shirts are similar in style.
I wouldn't say "their" style because I feel that the inanimate T-shirts do not possess a style; rather they were styled by a designer.
But as people tend to choose a style of dress for themselves it is common to hear:
My daughter and her friend have a similar style.
As this is possessive, I would suggest it is acceptable then to also say:
My daughter and her friend are similar in their style (of dress).