In the following dialogue:
John asks with "come with me" instead of "go with me".
Peter answers with "I would go with you" instead of "I would come with you".
Why is this?
The "with me" part of the dialogue is very important.
If I ask you, "Do you want to go to the zoo tomorrow?" and I am not planning to visit the zoo, then I could not use the verb come instead of go, unless I was the host of the event. (A zookeeper might say, "Do you want to come to the zoo tomorrow," but a friend who is not going along would not.)
However, if I ask you, "Do you want to go to the zoo tomorrow?" and I am planning to visit the zoo with you, then I can use come instead of go and the meaning will not change.
Sometimes the "with me" is explicitly stated (like in John's part of that dialogue), while other times it is unstated but implied (as in Peter's line of the dialog).
This is probably less about grammar and more about stylistics.
John could have said: "Do you want to go to the zoo with me tomorrow?" and it would mean the same but it sounds a little unusual. He could also have simply said "Do you want to go to the zoo tomorrow?" - the fact it is together would be implied.
He could also have said: Do you want to go with me to the zoo tomorrow?" if, for example, they were both heading the zoo and he was asking whether Peter wanted to travel together.
However, using "come" signals a sense of proximity and creates a warmer invitation since the emphasis is on going somewhere together, rather than simply going somewhere.
Peter could have replied "I would come if I had money". It sounds equally correct in English and has the same meaning as "go".
I understand your confusion. I'm a native English speaker and the first sentence in your question sounds wrong to my ear.
The reason is the what Janus Bahs Jacquet mentioned in a comment (to Laura's answer): Come indicates moving towards the speaker and go indicates moving away from your current position.
Of course, English changes. When I was in 3rd grade (1973) our English class had a section on common mistakes. One of the mistakes in our book was to confuse bring and take. The analogy is the same as come and go. "Bring" indicates to transport something towards the speaker while "take" indicates to transport away from the speaker. I remember laughing out loud when our teacher covered this, thinking that no one would make that mistake. Yet today I hear professional, national newscasters use "bring" in situations where I would use "take." So, apparently, the rules of accepted usage are changing.
I'm surprised to see so many answers on this page that say the first sentence is fine. It's painful to my ear, but it's possible that the meaning of come and go may be changing in a similar way as bring vs. take.
It's possible that your teachers or reference books are from my generation and they are teaching you rules that aren't always followed by younger speakers.
Without a doubt, I would ask Peter "Do you want to go to the zoo with me tomorrow?" (Unless I was already at the zoo, maybe thinking about doing it again the next day. They I would phrase the question using "come.")
So which should you use? You will find plenty of articles on the web that recommend the distinction I was taught, so I would consider that the safest approach. See:
http://www.eslcafe.com/grammar/confusing_words_come_go.html
https://dictionary.cambridge.org/us/grammar/british-grammar/come-or-go/
http://lingo-apps.com/come-vs-go-english/
In formal speaking or writing, I would recommend using the difference described in the web articles above. In informal situations, with younger speakers, you may want to try come instead of go, and see what sort of reaction you get.
EDIT: I was running some Google Ngram viewer queries and they made me realize this situation is delightfully more nuanced than I realized at first.
Because the example sentence in the original question ends with "tomorrow," I still think the 1st usage applies best. However, maybe the 2nd usage is contributing to the loss of distinction between these cases with younger people. I mean, there is some subjectivity to the idea of "immediacy."
For example, if my friend and I were sitting at home; I'm planning on going to the zoo tomorrow, but my friend is planning on going to a soccer game. If I want to change his mind; if I want him to "change his plans towards mine," so again the emphasis is on accompanying me, then again I might ask "Do you want to come to the zoo with me tomorrow?" In that situation, I would be just as likely to ask if he wanted to "go" to the zoo with me. That would really depend on the emphasis of my mindset at the time.
I think John wants to accompany Peter to the zoo tomorrow, so it is natural to use come instead of go.
Peter has not decided to accompany John. He says if he had money, he would go. so he said go instead of come.
I once tried to explain this as simply & concisely as possible to a non-English native friend of mine.
We arrived at this definition.
It depends on where the asker will be at the time of the event.
If the asker will be at the zoo, it's 'come' - as in 'come to where I will be at that time'.
If the asker will not be at the zoo, it's 'go' - as in 'go to somewhere I will not be'.
That simplest definition can be confused slightly if the asker is 'at that place now, but won't be later'
Asker is at a mutual workplace, colleague is currently at home.
"Are you coming to work tomorrow."
would be the form used, even if the asker was not going to be there tomorrow.
Another blur if you are both together right now & are both going to go to the same place.
"I'm going to the zoo, do you want to come with me? Sure, let's go to the zoo."
Though this one is really trading on the fact that both will be there by the end of the transaction.
Aside from those slight definition blurs, the simple rule can be used in almost all circumstances.
Even native speakers can break the 'rule' on occasion, but if you stick to the simple form, you will be correct 99% of the time.
