The second "little" creates an antanaclasis, a type of literary device, with the first "little," which is being used in accordance with one of its definitions as an adjective. The second "little" in this sentence is being used in accordance with one of its other definitions, one as an adverb, specifically an adverb that modifies the verb "did...know," that means:
- not at all (used before a verb):
He little knows what awaits him.
With that in mind, that coordinate clause essentially reads:
"...but not at all did I imagine it would come to this.