This usage presents a complex (one might even say muddled) combination of factual assertion about the way things were and a value judgement about the way things weren’t (but could have been).
The two interrelated statements are:
. . .when pioneering days were over, self-reliance [. . . .] remained the quintessence of many American women.
That which did not remain the quintessence of many American women was a conception of femininity that the author believes to be over-sentimentalized.
As your linked definition states, if something is sentimentalized, it is “present[ed] in an emotional way, emphasizing its good aspects and not mentioning its bad aspects”. The definition also notes that use of this word conveys a disapproving tone (making this use of “over-” odd and sentimentally(!) redundant).
In your question, you correctly infer that use of “rather than” in this sentence places “self-reliance” and “over-sentimentalized femininity” into opposition with one another. This is not merely an opposition of good and bad, but also of fact and fiction. Self-reliance is not only depicted as a positive trait, but also as the quintessential trait that the author believes these historical figures exemplify perfectly. It is then placed in opposition to the counterfactual reality that they would exemplify something else, while simultaneously placing the women themselves in opposition to a cultural force with which they had to contend: the idea that femininity and self-reliance are opposed!
All these nested oppositions are a lot for one statement to bear, but in the context of the whole essay it seems clear that the author means to say that these women were self-reliant; they rejected the pervading, romanticized conception of femininity as preclusive of self-reliance.
Remember that “sentimentalized” refers to how something is depicted, not how it is. It doesn’t mean that something was made sentimental, or even that it was made to seem sentimental (though the author’s somewhat sloppy use of the word might indicate undertones or pending shifts in its meaning). The only thing that’s sentimental is the way in which the sentimentalized object is viewed or presented, and when someone says a depiction is sentimental, they’re saying that it has undue focus on rosy ideals over harsh(er) realities.
Examples: People (and their lives) are often sentimentalized in eulogies, which are characteristically positive and focused on the interpersonal/emotional impact a person had. Consider this in partial opposition to a C.V., which is likewise positive but focuses on quantifiable acheivements in the real world (i.e. in relation to systems/institutions); or, perhaps, opposed in another way to an ad hominem attack, which is emotionally driven like sentimentalization, but characteristically negative.