Because such an auxiliary verb would cause problems.
If we compare "spoken Latin" to "written Latin", we're comparing two noun phrases. We are not comparing clauses. The participle "spoken" acts as a general modifier and does the same job here that an adjective typically does. We're still comparing two noun phrases when we compare "Latin spoken by the elite" to "Latin spoken by the masses".
This is good. The preposition "of" is happy to take a noun phrase as its object. We can account for everything, and the sentence holds together.
In contrast, "Latin is spoken by the elite" is a complete independent clause. It could be its own sentence. It doesn't resemble a noun phrase at all. The preposition "of" can't make any use of it.
With the auxiliary verbs in question, the preposition lacks an object and the independent clause lacks a transition. We can't account for everything. The sentence falls apart.