The past perfect of to stand is had stood rather than had stand, so editing your second sentence slightly to account for that we get:
If he had stood near the house when the wall collapsed, it would have killed him.
And as you suspected, this does sound a little jarring. Past perfect [had verbed] should be used to describe actions which were already complete when another action in the past occurred. [If he had moved closer to the building before it collapsed, he would have been killed.] In this example the moving was a completed action before the wall fell.
Thus the correct usage would be the first example you listed, in the past perfect continuous.
If he had been standing near the house when the wall collapsed, it would have killed him.
It is, however, not uncommon to hear the past continuous used in conditionals even when referring to a past condition. This is somewhat less common with the simple past, but again not unheard of.