(I'm speaking as an American, who is somewhat familiar with UK culture, but haven't been there myself)
I would use either one, without a strong preference for one or the other. In a more formal environment, "pounds" would be more appropriate (as with most slang). If in an informal conversation where someone else referred to it as "quid" already, using "quid" would probably be more appropriate (again, as with most slang).
I don't think it would be particularly odd to hear a non-native fluent English speaker (you appear to be fluent, or nearly so) hear this kind of slang. It would make me think they're a little more familiar with English and the UK than if they only used or knew "pounds". (which may be something you want to suggest or avoid, depending on whether you want them to treat you more like someone fluent in or learning English)