I have never heard him to say it
Now, I realize that the "to" is not supposed to be there. But why? Grammatically "say" in this sentence is an infinitive. Or it is pretending to be one. I am using other languages to get to the bottom of this and "say" ends up being an infinitive in each of them.