Please, help me to understand the usage of these modals better.
As I think, the most appropriate answer is
I wasn't able to drink coffee (come home late) when I was 10 years old.
because these situations are not about physical abilities (as something wrong with swallowing liquid or with legs :), but about external circumstances e.g. parents' rules, drinking coffee restrictions.
What's your opinion?