Defy in this case is taking the slightly less common meaning of "challenge" or "dare" (third at this link) rather than "resist" or "disobey." This usage of defy is always structured "defy [someone] to [do something]."
The "large number of people" refers to the corps.
The speaker is saying that if the entire corps (i.e., the whole army, or a large division of it) joins in the activities, the authorities will not dare to upset the whole group by declaring their activities illegal.