If you want to use the past perfect for the action happened earlier, the following sentence is grammatically correct:
I had dozed off before he finished speaking.
But I think as the conjunction "before" already tells you about which action happened earlier, there is no need to use the past perfect in the sentence unless you want to emphasize the first action. To me, the use of the past simple in both the clauses sounds more natural and easy on the ear. Look at the following sentence and find how you feel it.
I dozed off before he finished speaking.
As for the sentence under discussion, that is, "I dozed off before he had finished speaking", it's more natural and grammatically correct. As a matter of fact, according to grammar, you can use the past simple in the main clause and the past perfect or the past simple in the subordinade before-clause. There is no difference in meaning. So the following sentences are more common and correct grammatically:
I dozed off before he finished speaking.
I dozed off before he had finished speaking.
Source: John Eastwood Oxford Practice Grammar (Chapter 19, Review of the past simple, continuous and perfect).