We had [to go [to get a book]].
Preliminary point: most speakers treat stative "have" as a lexical verb, but some treat it as an auxiliary.
The sentence consists of a main clause (the sentence as a whole) and two embedded subordinate clauses, as bracketed.
"Have" is a catenative verb and the subordinate clause "to go to get a book" is its complement.
"Have to" is used in the sense of "must" to mean something is obligatory. "Must" is an auxiliary verb; but "have" is a main verb.
e.g., I must do it. (Subject + auxiliary verb + main verb + object)
= I have to to do it. (Subject + main verb + to-infinitive)
I will have to do it (Subject + auxiliary verb + main verb + to-infinitive).
"We had to go to ...
I think you may be confusing different uses of the same words.
As clever as I am, I couldn't repair my car.
This looks like an adverbial phrase meaning "Even though I am very clever"
He is as clever as I am
This looks like a complement.
The words are the same, but the intonation would likely be different, with different stress.