First the speaker says that women should be beautiful1. Then he says that their beauty would be distracting2.
Then the interviewer asks why someone would be distracted. There are women in the orchestra who would not be distracted by another woman.
Then the language becomes a little odd, probably since it's a translation.
It might be better translated as:
Appearance is something that doesn't continually hold your focus. Eventually you switch to the important part of the issue3 which is the conducting. After all there are women conductors.
I don't think that the quote is referring to someone tiring over years, but rather something that doesn't last during a performance. Women have proven their ability to be conductors and there is no reason to think that their appearance continually causes the orchestra to be distracted.
1. The important thing is, a woman should be beautiful, likable, attractive.
2. Musicians will look at her and be distracted from the music.
3. heart of the question