I think the difficulty here is purely the usage of the verb to spare when applied to time. Its relationship to the adjective is complex; technically, there are two separate routes into the language for different senses of spare. Even for those separate routes, it can mean opposite things. As an adjective it can mean surplus, or it can mean sparse - having more than needed, or very little. As a verb, it can mean to give, to save, or to withhold. It is simply a matter of usage to become familiar with its meaning in different usages. In a sense, most uses of the word could be seen as idiom or set phrase, simply because you have to know what it means in different contexts.
When used with time, it usually means excess as an adjective, or to share or give away as a verb. Your spare time is the time you have that isn't taken up by necessary things. If you can spare someone some time you are 'giving' that time to them. Thus, having time to spare means having more time than you need, much as you might have food to spare, money to spare ("spare a dime, fella?"), and so on.
This contrasts with a "spare meal" (which is frugal and limited), or the expression "spare the rod, spoil the child", which says that withholding corporal punishment will lead to a spoiled child.
Words with multiple contradictory meanings happen. My favourite example is the verb to cleave. If one cleaves rock, one splits it, but if one cleaves to the rock, perhaps while trying to make one's way along a narrow ledge, one is holding close to it. In the case of to cleave, the structure of the sentence usually tells you the meaning. With to spare, it is just a matter of knowing how it is used in different situations.