The original text is
they invented coins of different sizes
Is it OK if I say this instead?
they invented coins in different sizes
Yes, it's okay. You'll be understood, and I doubt many would even pause upon hearing it.
That being said, I think "of different sizes" is the more natural and common way to put it, or perhaps "of different size".
See this related English SE question: Different size or different sizes?
Also look at this Google NGram for "of different sizes", "of different size", "in different sizes", and "in different size".