When you say something like
I had been to the gym three times the previous month.
you're summing up, giving the cumulative number of visits, or expressing an aggregate experience as of the past reference time, the implicit temporal origo—you're not referring to action as repeated activity.
I had been there many a time but never saw her there even once.
A wooden paraphrase: As of some unspecified time in the past, your life experience included your having been there often. You are not referring to the repeated visits individually as repetitive actions. For that sense you would use past perfect progressive/continuous:
I had been to the gym often the previous month in order to burn off a few pounds.
I had been visiting the gym frequently during the previous month in order to burn off a few pounds.
The "facts on the ground" are identical. But with the past perfect, the month's total visits is emphasized, and with the past perfect continuous, the recurrence is emphasized.