If I were is used for unreals or hypotheticals - although (especially informally) "if I was" is often substituted for it.
However, If I was is correct for real, known or presumed conditions. Here, "were" feels wrong (perhaps a hypercorrection).
So if you can replace the word "If" with "Given that" or "Granted that" or "Assuming", then was is correct:
- He insists she was there. If she was there, we can find her on the CCTV footage.
- If she was there, why didn't she tell us?
- If he was there, she was (= must have been) there with him.
- I definitely saw everyone who passed by - so, although I don't remember seeing him, if he was there, I saw him (or must have seen him).
Going back to your sentence:
I would not have done ABC if I wasn't/ weren't sure
This reads to me like an unreal condition, making "weren't" at least as correct as "wasn't". Of course, a past unreal would more often be "...if I hadn't been sure", but "...if I wasn't/weren't sure" implies "and I am still sure now", whereas "if I hadn't been sure" could be read as suggesting "although I'm less sure now".