In the first sentence you use a present participle: some people also call it an active participle. You use it to talking about something that you are doing.
I had been studying maths - I am doing the studying
The second sentence uses a past participle: some people call it a passive participle. You use it to talk about something that is being done to you.
I had been studied by scientists - the scientists are studying me
This is probably not the meaning that you intend, so let's rule out sentence 2.
Sentence 1 is past perfect continuous: it says that you were doing something for a period of time (three years) before some event in the past (reaching level B).
Sentence 3 is past simple: you did something for an unspecified period of time, and you completed it three years before you reached level B.
Sentence 1 is closer to what you probably meant, but there is no real need to use past perfect continuous: you could just change sentence 3 from simple past to past continuous and add a for:
I was studying for more than three years before I reached level B
Note that the to after reached is not required.