There should be a comma in cases like this:
"I don't care," he said.
How about in cases like this?
I said(,) "Hello" to each of them.
The Grammabook site has a rule that says:
Rule 13c. If a quotation functions as a subject or object in a sentence, it might not need a comma.
Examples: Is "I don't care" all you can say to me?
I'm not very sure, though, if this applies to my example.