All grammar books teach us that we have to use Past Perfect for the action that happened before another action in the past, but I feel that it doesn't always make sense. For example, in this sentence, 'I had just two flights last year, and I'd had/I had 4 or 6 of them the year before,' we just state two facts about the past that are not connected. It's not a story. Moreover, it is clear which happened before thanks to the phrases 'last year' and 'the year before'. Is my gut feeling right? Or do we still need to use Past Perfect in this case?
"All the grammar books" are wrong. The past perfect is a useful technique for indicating when a action occured before the time that you are talking about. But there is no rule that says you have to use Past Perfect for the action that happened before another action".
In most cases where the past perfect is used, the past tense can also be used.
In your example, you have two events occurring at different times in the past. Naturally one occurred before the other, but there is no need for the past perfect tense.
You can use the past perfect to talk about situations in the past that arise from previous events:
I took two flights last year, and I'd taken five or six the year before, so you can imagine how I was building up air miles. So I decided to use them up by taking a holiday to India....
In this example we are talking about the events of last year, the earlier events affect the situation last year, so I used past perfect.
I took two flights last year, and I took five or six flights they year before. So you see that I am an experienced flier.
Here I have three times the first sentence is "last year" the second is "the year before" and these both have past tense. I've not use past perfect since I'm not talking about a past state. In fact I then go on to talk about the present.
If the 'proper grammar' feels clumsy, especially in the dialogue you quote here (you use 'I'd') then rewrite it to say what is important. In this example, you are using the pronoun 'them' to link back to the 'flights', so there is already a strong connection, and a conjunction would be appropriate.
Maybe say 'I had just two flights last year, compared with 4 or 6 of them the year before'?