In this answer, I have quoted definitions of could from the Cambridge Dictionary.
I could have done it
I would have been able to do it
The first two sentences have roughly the same meaning. In the first sentence, could has the meaning "used to talk about what someone or something was able or allowed to do". I am stating that, at the time, I had the ability to do it, but did not. The context would indicate whether this was because I decided not to, or because I was not given the opportunity or permitted to do so.
The first sentence might have a slight suggestion that I decided not to, and second sentence that I was not given the opportunity or permitted.
I could have been able to do it
This adds an extra degree of uncertainty: here, could is this meaning: "used to express possibility, especially slight or uncertain possibility". It suggests that I am uncertain that I would have had the ability to do it, had the opportunity arisen.