I think there is a scene with Humphrey Bogart where this exchange takes place:
Blonde: Would you mind buying me a drink?
Bogart: Why, yes I would (he says as he turns towards her and stares into her eyes)
Answering with yes I would will usually result in some further conversation since there is some ambiguity in the answer. Those additional conversations are sometimes best avoided.
The best way to refuse, is simply to say:
Sorry, I can't (and look the other way)
which implies you won't buy the other person a drink, but does not directly refuse them.
ADDITION: Of course if someone highly attractive to you asked, you could respond with :
Them: Would you mind buying me a drink?
You: I don't know. Would you mind if I bought you a drink? (emphasis on first you and smile)
and the conversation will take its natural course...