Yes, this is the common, uncountable, abstract sense of "damage". Since there is nothing that we can count, there is nothing to which we can assign individual sizes. There are no individuals to size. The large/small distinction simply doesn't make sense here.
Other distinctions do make sense. The much/little distinction represents degree in the same manner that large/small represents size. Severity can be represented by pairs like minor/major and gradations like slight/moderate/severe. There are, of course, many other polarities and spectra that make sense with uncountable abstracts, as do many absolute attributes.
Damage was done.
Much damage was done.
Little damage was done.
Major damage was done.
Minor damage was done.
Slight damage was done.
Moderate damage was done.
Severe damage was done.
Physical damage was done.
Emotional damage was done.
Legal damage was done.
Indescribable damage was done.
Like any language, English has quirks. One of those quirks is the word "little". It has several senses. For example:
- much/little -- degree
- big/little -- size
- a lot of / a little of -- relative portion
We don't want to dismiss the word "little" simply because we've dismissed scales of size and portion.