There are two versions of "You said it!"
One version idiomatically means "You said all that can be said on that subject." It expresses emphatic agreement -- the original speaker has covered everything you could have wanted to say. In this usage, there is no implication of impropriety.
"We haven't eaten in hours! I'm starving."
"You said it. Let's stop for pancakes in the next town."
The other version is an abbreviation of "You said it, not me", which does suggest that the comment would have been rude if you'd said it, and may not be a particularly emphatic agreement. For example, if your friend says, "Oh, I am such a fool!", you might respond, "You said it, not me." This expresses that you agree, but you weren't going to call him a fool to his face, and technically you still haven't -- you just agreed that he called himself a fool.