Why is it "come" rather than "go" ?
Let's see if we can confuse things even more...
Come and go are deictic when they refer to travel or walking, and usually they express the point of view of the speaker.
But when you are speaking with someone, if you use the word go, you are referring to a place that is not where you are and also not where the person is. With go, you are referring to a third place where neither of you are.
Imagine a phone call:
Hi, Jane. I am going to the library. (The speaker believes Jane is not at the library now).
Hi, Jane. I am coming to the library. (The speaker believes Jane is at the library now).
Shall I come to your place?
Shall I go to your place?
Both the sentences are correct grammatically; the use of these verbs depends on the context.
You use come to describe movement between the speaker and listener, and movement from another place to the place where the speaker or listener is. Look at the following sentence.
A to B: Shall I come to you or to your place?
It's the place where the listener is.
A to B: I want you to come to me or to my place.
It's the place where the speaker is.
A to B: I am going now to the post office to send a parcel. Do you want me to come to you from there?
You usually use go to refer to movement from where the speaker or listener is to another place.
A to B. I want you to go to Adam's house right now. I'll meet you there.
As for the sentences presented by the OP, you can use come if the listener is at his place (say his home,). However, you can use go if he isn't at his place (say he is in the market at the time of speaking to him).
You should consider the situation before using come or go.Come is used for movements to the place where the speaker or hearer is ( see your example) whereas go is used for movements to other places.
Let's go and see Peter.
Sometimes a third person becomes the centre of our attention. We use come for movements to the place where he or she is (was,will be).
He waited till two o'clock, but she didn't come.
Come over has one of the meanings to visit you at your house.
This is a bit tricky!
The word 'come' is taken in the context of 'who is listening to you'. If that 'listener' is with you, both work -'come', and 'go'.
For instance... if Mike and Tina both are in the same room, and Mike is going somewhere. Tina can say -
I am coming with you OR I am going with you.
But then, if someone is at some other place, 'come' is more convincing.
For example, if Mike is at his place and Tina is at hers, and Tina wants to go to Mike, she'd say...
goingcoming (over) to your place.
Here 'go' won't work. However, 'go' is possible but then it suggests that Mike is not at his place.
For example Mike and Tina both are at railway station, and Mike forgot his keys at his apartment. Mike has some wound in his legs. Tina can then say...
Hey, don't bother yourself. I'll go to your apartment to get the keys.