Independent IT consultant here, which means, I know the customer support hustle.
I don't think what you said is inherently rude, but neither do I think it would be a stretch for some to construe it that way. In some circumstances it might be taken as a little dismissive, or overly informal by someone accustomed to being treated with extra respect. I would probably save "you can go" for those people who it's safe for you to kid around with. It might sort of carry the connotation (even if jokingly) that the person needs your permission to go.
There's lots of ways you can end a support interaction, even enphasizing that you're busy, while still being unquestionably polite.
Before I suggest a few, bear in mind that inflection, facial expression, body language, etc, do matter, and anything at all can be made rude or polite depending on those.
That said: with a proper demeanor, I can pull any of the below off, and you should be able to to. Also remember: Fully half of being a service provider of any sort, especially in tech, is psychology. Part of providing good service is, you want them to feel like you're there at their service. Remembering that is probably more important than choice of words.
I would say you want to communicate three things to wrap up a support call in a way that leaves the client satisfied: first, you say something that confirms the task is finished, then an inquiry to make sure they agree it's done and don't feel you're leaving them still needing any help. Then, finally, let them know you're available if they need you again. This is how I do things.
So, the first two would be something like:
"I do have another appointment waiting. Is there anything else I can help you with?" (Note how this one is emphatic, even businesslike, but not rude.)
"Ok! That's all working now. Have we gotten everything you need taken care of?"
"I do have to rush off. Are we all good?" (Note: "Are we all good" is a very informal turn of phrase, but, in all but very formal situations, can probably be gotten away with, especially if you know the person casually and can already speak somewhat informally with them.)
Or just combine the first two thoughts with a succinct, "Ok, is there anything else I can do for you?" That tells them "I think we're done, but I want to make sure you're satisfied."
For the third and closing part, be sure to conclude with an emphatic, "Please call me immediately if there's anything else I can do for you" or "call me immediately if you think of any questions." This should be carried out with an air like you consider it your job to be at their disposal. Even if it's not, people like feeling that way. The key component in all of these, you might notice, is that you're placing their needs as the most important priority, no matter what. I believe that actively communicating that is the key thing to bear in mind.
I do have one more thought about your interaction. Since you're posting this here in "English language learners", I assume there's something of a language barrier involved. I think a lot of people will be a little understanding of people from other cultures or non-native speakers perhaps coming across rude simply out of unfamiliarity with what's (somewhat arbitrarily, really) considered polite or not in different places, and he may have been trying to be helpful to you–not saying you were rude because he took offense, but rather, just out of understanding that as an English learner you may not have all the Ps and Qs (that's a colloquialism for good manners) yet, and he may just have been trying to do you a favor by offering you some guidance, letting you know that other people might potentially interpret it that way. It could depend strongly on his inflection.
I would say if you're very clearly still an English learner in your spoken interactions, if he was sincerely offended, he'd be the rude one for not cutting you some slack that you deserved.
I learned this the hard way, visiting England at 14, and seeing the expression on my elderly and very prim hostess's face when, during a visit for tea time in her living room, I joked around with a fellow tourist using a word I'd just heard for the first time that day: I called someone a "wanker". Suffice to say: it was a faux pas!