If we can say: He is considered (to be) smart. We consider him (to be) smart.
By the same token, I have also heard: We think him (to be) smart or think him dead.
It is also correct to say: He is said to be smart.
But, Is it correct to say???: We say him to be smart.
For some reason it appears to follow the above structures, but sounds very strange.