So I remember a lot of the time, outside the summer months, being chilly. You went around with two, even three jumpers on, and your jeans felt cold and stiff.
I wonder, why is the outdoor chilly in the summer?
In this context, outside is not referring to the outdoors. It is referring to months that are not summer. These months are considered "outside" of the group of months labelled under summer
It conveying the meaning of "besides the summer months" or "not in the summer months".
Another example would could be
"Outside the technical subjects, my favorite class is creative writing".
This would imply that the speaker generally likes most technical subjects a lot. But besides those subjects, her most liked subject is creative writing.