I understand your confusion. I'm a native English speaker and the first sentence in your question sounds wrong to my ear.
The reason is the what Janus Bahs Jacquet mentioned in a comment (to Laura's answer): Come indicates moving towards the speaker and go indicates moving away from your current position.
Of course, English changes. When I was in 3rd grade (1973) our English class had a section on common mistakes. One of the mistakes in our book was to confuse bring and take. The analogy is the same as come and go. "Bring" indicates to transport something towards the speaker while "take" indicates to transport away from the speaker. I remember laughing out loud when our teacher covered this, thinking that no one would make that mistake. Yet today I hear professional, national newscasters use "bring" in situations where I would use "take." So, apparently, the rules of accepted usage are changing.
I'm surprised to see so many answers on this page that say the first sentence is fine. It's painful to my ear, but it's possible that the meaning of come and go may be changing in a similar way as bring vs. take.
It's possible that your teachers or reference books are from my generation and they are teaching you rules that aren't always followed by younger speakers.
Without a doubt, I would ask Peter "Do you want to go to the zoo with me tomorrow?" (Unless I was already at the zoo, maybe thinking about doing it again the next day. They I would phrase the question using "come.")
So which should you use? You will find plenty of articles on the web that recommend the distinction I was taught, so I would consider that the safest approach. See:
In formal speaking or writing, I would recommend using the difference described in the web articles above. In informal situations, with younger speakers, you may want to try come instead of go, and see what sort of reaction you get.
EDIT: I was running some Google Ngram viewer queries and they made me realize this situation is delightfully more nuanced than I realized at first.
- If a friend and I were sitting at my house and we were making plans for the next day, then I'd still recommend the come vs. go distinction described above.
- But there is a different usage. If I were on my way to the zoo alone, but I saw a friend and I wanted to see if they wanted to accompany me right then and there, then I might ask, "Do you want to come to the zoo with me?" I suspect this usage is a little idiomatic. I'm really inviting the friend to come to me, and then we will go to the zoo together. I'm still thinking this through, but I think I would only use this expression for an immediate situation: When we are going to the zoo right then and there. When I could actually extend a hand to the friend, as if to draw them towards me. In a sense, the emphasis is on asking the friend to "accompany me" (hence use of "come") and where we will be going is secondary. If you rephrase the first sentence as "Do you want to come with me to the zoo?" then the "accompany" meaning of this usage is easier to see. This may be why the answer from J.R. feels that the "with me" part of the question is important.
Because the example sentence in the original question ends with "tomorrow," I still think the 1st usage applies best. However, maybe the 2nd usage is contributing to the loss of distinction between these cases with younger people. I mean, there is some subjectivity to the idea of "immediacy."
For example, if my friend and I were sitting at home; I'm planning on going to the zoo tomorrow, but my friend is planning on going to a soccer game. If I want to change his mind; if I want him to "change his plans towards mine," so again the emphasis is on accompanying me, then again I might ask "Do you want to come to the zoo with me tomorrow?" In that situation, I would be just as likely to ask if he wanted to "go" to the zoo with me. That would really depend on the emphasis of my mindset at the time.