Imagine somebody is trying to achieve something, and explains you what the task is. At some point, you need to make clear what it needs to be done, since the person talking to you doesn't seem to understand what you previously said.
Is it rude to start a sentence with "First off"?
As another example, suppose somebody is telling you that you said something, when you didn't. Would be rude to answer with "First off, I didn't say that."?