I really think that I should use "the baby can't eat BY himself" to make it clear, but my friend argues that the first phrase would still be correct and mean the same thing as the second phrase. But I am still not sure, "can't eat himself" just sounds wrong to me. By the way, me and my friend are not native english speakers.
It should be, "The baby can't feed himself."
While most people could properly infer your meaning if you said, "The baby can't eat by himself," that sentence could also mean that for some reason, the baby needs to eat with other people in attendance, not necessarily because he is incapable of feeding himself.
You are correct that, "The baby can't eat himself," is absolutely the wrong thing to say. This sentence means that the baby cannot consume his own body as food. This is a very odd thing to say.
The sentence could be idiomatic without by but it would depend on context and on the verb in question, and its transitivity or intransitivity in context.
Who ate the last piece of beef jerky?
-- I just had dental work and can't chew myself, so it wasn't me.
Are you climbing up to the top of the Washington Monument with us?
-- I've sprained my ankle and can't walk myself, but the rest of you should go on up without me.
These sentences with the reflexive pronoun could be paraphrased:
As for me, I just had dental work and cannot chew, so ...
As for me, I have sprained my ankle and cannot walk, but ...
The same idea could be expressed this way:
Me, I just had dental work and cannot chew, so ...
Me, I've sprained my ankle and cannot walk, but ...
So if you were to say
The baby can't eat himself...
The meaning would be:
As for the baby, he cannot eat ...
The reflexive pronoun there, without by, would not mean "alone, unassisted".
With transitive verbs the meaning would be "alone, unassisted":
The baby cannot bathe himself.
The baby cannot feed himself.
The baby cannot dress himself.