I've read some posts differentiating come and go on the site, however, there's a case which hasn't received that much attention, which is when the speaker and the listener are both at the same place, and they are going somewhere.
A native speaker said (here), and I quote,
"Keep in mind you have to use the perspective that makes sense in context. If my friend and I are thinking of a trip to Europe, I would say
Let's go to Europe.
and not
Let's come to Europe
... However, I can ask my friend in the US:
Do you want to come with me to Europe?
because I visualize that person with me as we move toward a location -- she's "coming along" with me."
I honestly don't see the difference between the two cases. How is "Let's come to Europe" wrong, but "Do you want to come with me to Europe" right? The difference here is just the word with. But if it's all about whether you visualize your friend coming with you or not, a word shouldn't change anything here. You can still visualize your friend coming with you to Europe while saying "Let's come to Europe!"
To sum up the question here: by his logic, I don't see any problem with the sentence: Let's come to Europe! Why did he say it's wrong but state that "Do you want to come with me to Europe" right?