The Lexico definition is unambiguous, though I'll admit that it's a bit wordy and hard to understand for learners.
As you said, the definition is:
"of" expresses time in relation to the following hour
If we use the actual values in the phrase "a quarter of three," then this means:
"of" expresses one quarter hour in relation to the following hour of 3:00
I've emphasized the following hour because this tells us that we are talking about the quarter-hour that precedes 3:00, which is 2:45.
A clearer definition could be:
"of" expresses a short time span before an hour
Note that even in AmE, this is a regionalism - it's common in the Northeast and rarer elsewhere. It comes very naturally to me because I grew up using it, but when I moved to Canada, I had to train myself not to say "five of nine" to mean 8:55 because nobody here understands this expression.