As has already been answered, the difference between the two statements is the tense - one is looking back to the past, one looking forward to the future.
Saying "I don't know if I could have done it", after seeing someone else's performance, is a sort of comparison - you are saying that you doubt you could have done what they did.
Saying "I don't know if I could do it" instead ultimately expresses the same self-doubt, but instead of putting yourself in their place and imagining you doing it in place of them, you are asking if you could also do it.
For clarification, when I say that these express "self-doubt", neither are really self-deprecating comments in this context, but meant to express admiration for the other person's ability.