In the US, I hear tee used stand-alone more often from women than men, but maybe that's because I'm more likely to be talking about clothing with women. It's not as common as t-shirt, but it's not at all strange for me to say, for example, "do you know where my purple tee is?" It's usually clear from context what I mean—for example, if I'm digging through the laundry without a shirt on while asking the previous question or if one of my kids asks about the weather and I say "I think you'll be fine in just a tee."
It's hard to get a sense of spoken patterns from a written record, but there are a lot of blog posts (like this one) and tweets (like most of these) that mention things like "jeans and a tee" or "my favorite tee" without using the full tee-shirt, so I don't think I'm alone in sometimes using the shorter form.
Also, in some places tee is countable, so it would be unlikely to be confused for the beverage—for example, I would say "I got you a tee" but "I got you some/a cup of tea". In places where the beverage is also countable, it would be more important to rely on context (especially if the tea is green or the tee is hawt!).
You don't need to use the shorter form, though; it's always safe and acceptable to say t-shirt in full.