This is a fairly common question in English and is confusing until you understand the logic behind it.
In English you imagine yourself in a particular place or point of view, and then describe "come" or "go" from that point of view.
I will go home at 5pm (my point of view is wherever I will be at 5pm)
I will come home at 5pm (my point of view is at home)
In your example, the speaker takes his point of view from their current position, as if watching them get closer. It is slightly warmer, as it sounds nice to say that you'll come visit someone, but it's also logical since that's where they actually are.
It can be fine to use "go" instead, although it helps to first establish point of view:
We were all just sitting around the house so we decided to go down and see you!