"swear off" means to promise to abstain from something. It isn't really a correct way to say that someone was removed from a job or position.
Some better ways to say that would be "he resigned" if he left the job by his choice, or "he was dismissed" if he left the job and it was not by his choice. If he is giving up that line of work, or giving up working altogether, you could also say "he retired".
If you don't want to say whose decision or was, or if you don't know, I don't know of any analogous phrase that means exactly the same thing. You might have to reword it as something like "he no longer works there".