If he's talking about something that has already happened, then the correct wording would be: If I did that.
If he's talking about something that could have happened in the past, then the correct phrasing would be: If I had done that.
Wikipedia has a good article on English conditional sentences. When expressing a hypothetical situation, "would," "had," and "were" are used instead of other verb forms, e.g.:
I knew he would do poorly if I were there, so I didn't go.
I knew he would do poorly while I was there, which is why I went.
In spoken English, people often find this phrasing awkward. For example, this sentence is correct:
If he were there, things would have been different.
However, many people would change that phrase to the following:
If he was there, things would have been different.
Even though the second sentence is incorrect, it is more comfortable for many English speakers because the phrase "he was" is far more common than the conditional "he were." Also, "he were" is incorrect in every instance except a conditional sentence. Many English speakers do not know (or do not care) about this particular case of subject-verb agreement.
So, from the example you've given us, the "that" which your speaker is talking about ("if I did that") is probably the thing that he chose not to do.