From the context it seems you're correct that Charlie is referring to sex. This can be inferred from a few contextual clues. When Joanne says that "everything...is on the line, including your manhood", Charlie interprets that as Joanne saying he is not manly, and because of that he won't be having sex again because he believes sex to be a "manly" thing to do. There's also the added connotation of manhood being a commonly used euphemism for a man's genitalia, which reinforces that idea.
There's also the fact that after Charlie mentions that he won't be getting it twice tonight that Joanne implies that he can't count, meaning that they will be having sex more than the once they've already had. It's a playful comment which adds to the idea it's about sex. There's also the fact that they don't refer to sex explicitly, which is common with people when they're talking about sex, regardless of how well they know each other.