Be careful about "don't bother", it can be interpreted as actively hostile.
It's about like when you say to your other half, "I'm going to spend the rent for the next 3 months on a new motorbike", and she (sorry, it's usually "she" in this context) says "Fine!" when she really does not mean "fine", what she means is "I'm obviously not going to be able to stop you doing whatever immature selfish thing you like, but if you do, then our relationship is over."
In this case, "Don't bother" means "It's obviously far too much trouble for you to just perform this minor courtesy for me, and I clearly cannot rely on you to pull your weight in this relationship, so please be aware that our relationship is on shaky ground as of now."
So if someone offers to wait for you, and you say "Don't bother", it can be interpreted as a rude rejection of a courtesy offered, and the person asking would spend considerable emotional energy wondering what it is that he or she has done to upset you.
Better would be:
"Thanks, but you go on ahead, I'll catch you up later."
Then there is nothing in what you said that indicates that there is any sort of obligation that you need to be waited for.
Even saying "thanks, but you don't have to" is using the language of obligation and duty ("Have to") and can cause awkwardness.