I read this passage in I heard you paint houses:
The night before my testimonial dinner, Russ and I had a sit-down with Jimmy. We sat at a table at Broadway Eddie's, and Russell Bufalino told Jimmy Hoffa flat-out he should stop running for union president. He told him certain people were very happy with Frank Fitzsimmons, who replaced Jimmy when he went to jail. Nobody at the table said so, but we all knew these certain people were very happy with the big and easy loans they could get out of the Teamsters Pension Fund under the weak-minded Fitz.
I'm not sure what it means here. I have seen 'say so' in dialogues like:
A: Hey You said you were afraid of dogs, right?
B: I didn't say so.