If it is your friend's party, then you say
Yes, I'm coming (to your party)
No, I'm not coming (to your party)
regardless of where you both are, since you are talking directly to them
If a third friend is asking you about your friend's party, then you say
Yes, I'm going (to their party)
No, I'm not going (to their party)
regardless of where you are