You're not wrong. Quite the opposite, you're absolutely right in thinking that there is a problem with it. You see, phonetically, must've, when you say it fast, sounds very similar to the way you pronounce must of. People just write it as they hear it with no attention at all paid to the rules of orthography. The same thing is also true for things like could've, would've and should've where people commonly misspell them as could of, would of and should of.
And because a lot of native speakers never took the time to study English grammar properly (that's the sad truth), they often make this, I'd say, rather stupid mistake. No offense, but the term dumb would probably be a better word to describe this situation. Though some might disagree, this has really nothing to do with being informal either. That kind of writing style should make any person with at least some basic high-school education or a passing knowledge of English grammar cringe.