Basically, if the comparative is used as either the subject or object of the sentence, it's used as a noun - i.e. if you can remove that part of the sentence and get it by asking "Who" or "What".
In this case we have:
Q: Who/What do I like?
A: The smaller of the two.
An example where this cannot be used:
The first book is smaller.
If you would to remove "smaller" form the sentence and ask:
Q: Who/What is the first book?
"Who" doesn't make any sense, while "What" makes some sense but it's used in a different way.
Smaller describes the book, not itself. If the answer was
the smaller it would not fully answer the question, because it would just raise another -
The smaller of what?. The answer needs to have meaning by itself.
Side not on this example. If there was a text before that clearly defines what
the smaller is, we could use it. E.g.:
The first book is the smaller of the two. I am going to read the smaller.
the smaller clearly means the first book.
Hope this clarifies things a bit, articles can be tricky, you learn them mostly through exposure to the language. :)