That sentence is wrong: it should be 'two scores of books'; you need the plural there.
The 'of' is used because 'score' isn't exactly synonymous with 'twenty', rather, it means a set of (about) twenty things. Therefore, the word 'of' is used, just like you would say 'two sets of books'. Another example: you say 'hundred books' but 'hundreds of books'.
'dozen' also means a set of twelve, but it seems to have evolved into a proper cardinal numeral. Therefore, you say 'five dozen' instead of 'five dozens', and omit the 'of'. 'He bought five dozen bananas.' would be correct.