Formally, it is the same prefix "in-".
However, there are four different spellings that help pronunciation.
The rule only depends on the first letter of the adjective:
L — use "il-": illegal, illiterate;
P, M, or B — use "im-": improper, immediate, imbalance;
R — use "ir-": irresponsible, irrational;
other consonants or vowels — use "in-": indifferent, innaturalnumerable, inadequate;