Why do we go from using the prefix un- in unstable to using the prefix in- in instability? Why can't we just keep using un-?
According to http://www.worldwidewords.org/qa/qa-unv1.htm words with a Germanic root use the un- prefix and words with a Romantic root use the in- suffix.
There are also the prefixes a- as in asexual and non- as in nondescript.
Unless you're a professor of linguistics and etymology I think your best bet is to use un-, which is the most common prefix, and learn which words use a different prefix on a case by case basis.
EDIT: I know this doesn't completely answer your question for why unstable turns into instability, but the truth is that although there is a general rule for which prefix to use, there are so many exceptions that it's pretty much arbitrary (without reason).