Does this mean that many people have that opinion? Usually what comes after the "of" denotes the thing that someone else, mentioned before, formed his opinion of.(sorry confusing explanation) As in "That is YOUR opinion of X." So to avoid confusion would the above sentence have to be rearranged to something like "That is many people's(because many's doesnt exist) opinion." or better "Many people have/share that opinion"?.
The sentence is a shortening of this:
That's the opinion of many people.
Another way to phrase it would be this:
Many people hold that opinion.
People is generally left out because it's understood to be the object of that prepositional phrase.
As for the second example I put above, another way to phrase it is this:
Many hold that opinion.
where again, many is assumed to be about people. For me, a native English speaker, it wouldn't generally make sense to me to have an opinion on "many" (as in, my opinion on "many").