The problems with your sentence #2 are not so much grammar problems as problems of logic or idiomatic usage. You notice that the CGEL quote uses "living," a present-tense verb, to describe "at that time," presumably a time in the past. This could be understood as a shortening of:
Just about anyone who was living in the area at that time is at risk.
So your example could perhaps be understood as a shortening of:
The man who was stealing our computer last night was caught just now.
... but there's a difference. Stealing a computer is something you do over a very short time; living in an area is an ongoing state that describes a broad time. The CGEL source can use the present participle because they're describing this ongoing state; stealing a computer is a temporary activity.
By the way, your third sentence needs "The man who had stolen...", but perhaps that's just an accident.