- He was waiting for me when I
- He had been waiting for me when I arrived.
The first sentence means that when (at that moment) you arrived, he was there waiting.
The second sentence means he started waiting for you some time before you arrived.
Simple Past Continuous is the more natural form for OP's exact example, because it answers the question What was he doing at that time? (i.e. - at the precise moment in time when I arrived).
Past Perfect would answer the question What had he been doing before that time? - with the strong implication that whatever he'd been doing was a "completed" action by / before then.
There's a little bit of "semantic wiggle room" in the exact context, since we could say the act of waiting was still ongoing at the precise moment I arrived (or of course we might say it ended precisely then). But in practice, native speakers would usually only use a Past Perfect form here if additional context made it obvious the speaker was thinking in terms of "activity prior to arrival", rather than "status at time of arrival". Thus...
1a He was waiting for me with a huge grin on his face when I arrived
2a He had been waiting for me [for] over an hour when I arrived
Both of those versions are fine, because the choice of tense form matches the highlighted adverbial elements in each case (but they would both be *extremely unlikely / awkward if the verb forms were reversed).
It might be worth pointing out that the "unusual" form...
1b He had been waiting for me with a huge grin on his face when I arrived
...would force the audience to suppose that he was some kind of "happy idiot" who'd been continuously grinning for some time before I arrived - a ridiculously contrived situation, since we'd naturally be expecting that the reason he was grinning was because he'd just seen me arriving.