He said that he was having dinner when they knocked on the door.
He said that he had been having dinner when they knocked on the door.
Your first example he was having is called past continuous tense and expresses an unfinished action in the past that is still going on at the moment of speaking. Sometimes it can be interrupted or simultaneous with another past event like in your case. In other words the action of having dinner is interrupted or simultaneous with the action of knocking on the door. However, the past continuous tense has some more uses, some of them are listed in the link I provided.
On the other hand, your second example he had been having is called past perfect continuous and expresses a duration of time before something in the past. In your example, the action of having dinner was finished before the action of knocking on the door.
This said, your examples have quite different meanings.