One of the uses of the verb would is to talk about habitual action that took place in the past.
OED says of will:
27. (Cf. 8.) Was (were) accustomed to; used to.
...
1470–85 Malory Morte d'Arthur ix. xxxvii. 399 Euery day syr Palomydes wold repreue sir Tristram of old hate betwixe them.
...
1847 Thackeray Vanity Fair (1848) xxi. 181 The girls would ask her..for a little music, and she would sing her three songs.
When native speakers use would as in your example, we are using it as both a conditional and past tense, and this is as it should be, since would is the past tense of will. The condition is fulfilled in the past, and it is will's original implication of intention that provides the sense that the action is habitual.
I am venturing far from grammar here, but perhaps one of the reasons for your observation:
"the more I hear it and use it, the more it "feels" like conditional..."
...is that you are beginning to think in English, and that some of historic sense of the verb will is starting to influence that thinking. Consider that if a British person in 1594 CE said:
Husband wants me home, but I would go to market...
...she was expressing the thought: I firmly intend to go to market. Would is not just an auxiliary that puts go in a future tense or makes the "going" conditional. Even today, when we use would to talk about the past:
She's gone, but in the past, I would gather flowers for her...
...the statement has a greater affect than:
She's gone, but in the past, I used to gather flowers for her...
Used to merely expresses that the action was habitual and in the past. Would implies also that the speaker actively desired the action in preference to other actions.
Historical note
Originally, this usage of would to describe action in the past might also imply some of the original sense of will as intending or desiring, as in one of the OED's earliest citations above:
1470–85 Malory Morte d'Arthur ix. xxxvii. 399 Euery day syr Palomydes wold repreue sir Tristram of old hate betwixe them.
In this passage, Malory describes the imprisoned nobleman Palomydes and his daily reminders to his fellow prisoner Tristram of the enmity between them; yet, when Palomydes sees that Tristram is ill, he comforts him instead:
But when syr Palomydes sawe the fallynge of sekenesse of syr Tristram, thenne was he heuy for hym / and comforted him in alle the best wyse he coude.
The use of wold here describes both past action and Palomydes's will to continue haranguing Tristram: and then, seeing that Tristram is ill, he no longer desires to remind him of their past conflict. This sense would have been clear to a reader in Malory's time. Malory could have used the simple past repreued here, but may well have chosen wold repreue for this reason.
As centuries passed, this sense of intention or desire in most such usages was lost, and in most modern uses like the one in this question would connotes only used to.
