Modal verbs can be complex and seemingly irrational. However, I think I can explain this one.
"Need/need to" is a semi-modal, not a full modal. In the first sentence, it acts as a full modal (with the negative particle "not" added directly to the modal verb), where as in the second example, you need the auxiliary "do" to create the negative.
This changes the position of "not" in the sentence, and as word order is particularly important in English, it changes the meaning. Compare:
You needn't come = what you need to do is not come.
You don't need to come = there is no need for you to come (but a possibility to come still exists)
You can apply the same to must/mustn't (full modal) and have to/don't have to (quasi-modal) to understand why the affirmatives appear to mean the same as each other, but the negatives mean something different from each other.
Anyway, if what you need to do is NOT come, then the adverbial phrase "unless you want to" is clunky and awkward and perhaps, as you said, wrong.
As for the third sentence, well...
You need the word "to" for the omission of "come". "to" is the identifier that the word "come" is missing (to avoid repetition). In the sentence You can come if you want. there is no confusion, and so it is perfectly acceptable, as is You can come if you want to.