Imagine you are at the doctor's office and at the end when you are done you are leaving the office and the doctor comes with you to the exit door and you want to say thank you to doctor and you say "doctor I am leaving please do not bother yourself by staying there. Ok it might take half a minute for me to be gone from the entrance of the door, okay? so there is no need you stay there for half a minute, probably you have something to do and that is very kind of you to stay there until I am gone. What can I say instead of saying "do not bother yourself doctor" ? Leave me here and I have gone in minutes, do not waste your time by staying here and showing respect to me. Close the door and go inside please" .
You are right that this needs to be handled with a bit of care not to appear impolite. The doctor is taking the time to show you out, and it would be rude to decline his hospitality in an abrupt way. It would also be rude to press on his time by dawdling.
Therefore the best thing is to be honest: "Don't wait for me doctor, it always takes me a couple of minutes to put my coat on!" Or even, "I can see you are very busy, so let's say goodbye here, thank you again."
In English we also have an idiom, "I'll show myself out", which arises in this scenario. It used to be proper to escort your guests out of your house personally. However, you should be careful when you use it, because it is often used sarcastically.
As a native speaker, I would probably say something like, "I'm all set, then, thank you," or "Thanks again." You cannot really tell the doctor to go away without being rude, but you can make clear that you need no further assistance/company.