Verbally, people do this all the time, you're right about that.
I think whether it would (it's?) allowed in written speech depends a little on the thing being written. I wouldn't want to use that contraction in anything that was at all formal, but writing a personal letter to someone or relating dialog in a story of some sort, I would use the apostrophe that way.
The idea of a contraction (you have this question labelled 'possessive', but what we're actually discussing is contractions) is to use the apostrophe to denote that something has been left out. I've been known to use "I'd've" as a contraction for "I would have", which is also a way that some people speak.
I think saying that 's always denotes possession is a little too restrictive.