I think, grammatically, the 'even more so' refers to the amount that the speaker would not vouch for whatever it is. The 'much less so' refers to the amount of whatever it is being vouched for; since he is NOT vouching for it, they are both supposedly emphasizing the fact that he does not vouch for it in winter, and the amount that he would not vouch for it is increased in the summer.
From the standpoint of actually speaking about it, however, the first is less clear. The "even more so" leads the speaker to believe that something has been referred to and this phrase indicates more of it, but the thing referred to is a negative thing. So there is a confusion, momentary or not, whether the speaker is recommending more of something or has more negative recommendation for it. The second form is, therefore, clearer.
And neither of them are very good. Both of the latter phrases refer to an amount of something, whereas the former phrase doesn't have an amount to which to apply it/them. So it isn't a very good way to say whatever the speaker is attempting to say regardless, in my opnion.
EDIT: English, at least American English, tends to have phrases like this that are supposed to indicate an increased strength or amount of things that sometimes don't sound like what they're supposed to mean. One of the worst in this regards for non-native speakers, it seems to me, is "I could care less". In fact, the phrase the speaker is attempting to quote is "I could not care less", but the former phrase is used all the time to mean the same thing, even though its translation is the exact opposite. Both phrases mean the speaker is professing not to care about whatever it is, but it's hard to figure that out by a translation of the phrase used.