The last part of the sentence is a condition ("if a demolition company..."). That condition did not happen; it is counterfactual, and it refers to a past time frame.
In this type of conditional statement (called the third conditional), the condition should be in past perfect and counterfactual result is stated with "have", as in present perfect (this is called Conditional Perfect). So the sentence should better be
(...) no one would have noticed if a demolition company had destroyed it and drove it away in trucks.
The quoted sentence, with past simple instead of past perfect in the condition, is of the second conditional type, which refers to a counterfactual condition at the present time frame. In this case it would still be understood correctly from the context, but the third conditional is a better choice.