You're mostly right, or entirely right, depending on who(m) you ask.
I don't remember the name of the man who I talked on the phone
Is incorrect. That use of talk requires a preposition. It could be to, with, about, over, etc. "The man who I talked" is incorrect.
In your reasoning about the answer, you ignored the possibilities with whom. To people who are strict about English grammar, the correct answer must contain whom, because it's the object, not the subject in that sentence.
Many English speakers have decided this rule isn't important, and don't use whom at all. For them, the correct answer would be the one you gave: "...the man who I talked to on the phone." Other English speakers sometimes use whom, but don't remember the exact rules; they might remember that we often use whom when it follows a preposition. For those people (and for the strict grammarians) the correct answer is 3, "to whom."