Hitler himself related to me in 1930 – when we were speaking about his family relationships (in connection with a blackmail threat), ‘Even as a 10 or 12-year-old kid I always had to go late at night to this stinking, smoky dive. Without being spared any of the details, I would have to go to the table and shake him as he looked with a blank stare.
Source: Martyn Housden: Hitler: A Study of a Revolutionary?, p. 169.
I assume that using would ("I would have to go") indicates the past tense. Can I replace "I would have to go" with "had to go" without a change of the meaning in the sentence? Is the reason for this usage clearly stylistic: the author does not want to be repetitive.