My textbooks suggest that the only correct option is the one in the past perfect continuous because the past perfect continuous means an action ("planning") which lasted up to the later moment which is in the past simple tense ("I went out"). The past continuous, however, ("I was planning") is wrong here because it is used when one wants to say that an action was happening at an exact point in time - like 5 pm or when Mother was making lunch.
Which of the following sentences is considered grammatically correct and why?
"What was I planning to do before I went out?"
"What had I been planning to do before I went out?"