I think "so" in "you're so not cool" emphasizes the whole sentence.
Whereas, "so" in "you're not so cool" emphasizes "cool", meaning "very cool".
I am not sure whether I'm right.
So, What is the difference between "you're so not cool" & "you're not so cool"?
In short: you're correct.
Long-form: The use of "so" in
You're so not cool.
is extremely idiomatic, and informal. It's used to emphasize the degree to which "you" are "not cool". The result is a superlative statement, in a way, declaring the level of "not-cool-ness" to be extreme, and beyond measure. Verbally, the word "so" would be strongly stressed.
Compare:
You're not cool.
You're so not-cool [that I'm at a loss for words to express myself further].
This construct would be well placed in the film "Mean Girls", delivered as an insult with a dismissive and exasperated affect.
Conversely, placing "so" after "not" is much more thoughtful, (though still critical.) In this case the sentence suggests that the listener has an overly-inflated ego, or perhaps that the speaker formerly thought that the listener was once a "cooler" person than they are now perceived to be. The "not" is used as a polarity switch, and "so" is used as a modifier of "cool".
Compare:
You're so cool.
You're not so cool [as you think you are/as I thought you were].
This phrasing would likely show up in a light comedy scene, as a form of criticism, but lacking malice. It could be said between friends, and be at least forgivable, or maybe delivered and taken entirely as a joke, or "ribbing".
One may notice here that, as is common in English, both examples are used as shorthand to express a longer idea, which I spelled out in [square brackets], making them both "jargon", and incomplete in themselves.
Citing my source: I'm a native speaker of American English, raised in California, with some college, several years of customer service communication, and an author for a parent. Practical linguistics is a hobby.
