We all know that fuck means to have sex, so what does it mean when a speaker says fuck you?
If fuck me means the speaker is asking someone else to have sex have with them, then what does fuck you mean?
The Original Poster seems to be inquiring about literal meaning of the sentences "fuck me" and "fuck you". The illocutionary force of these sentences is a different issue altogether. This post then is about the grammar of the Original Poster's examples, and how this affects what they literally mean.
My own example, sentence (1) is clearly an imperative, where the phrasal idiom fuck off literally means something like go away. Sentence (2) on the other hand seems to be a case of what some writers refer to as a "subjunctive". It is grammatically similar to phrases such as:
As with imperative constructions, these constructions use the plain form of the verb. Notice that the saying is God save the queen not saves the queen. Also notice that when the verb is the verb BE we see the plain form not a present tense form such as is. However, these are not imperative constructions. When we say peace be upon you this is not a third person imperative such as students unite! We are not giving an instruction to someone called "peace", or to peace itself. The sentence means something like May peace be upon you.
Rather, these sentences are invocations where the act of saying them is presented as an exhortation that will make them hopefully occur. Where the Subject of the sentences is not overtly expressed, it can be thought of as being God, fate, the universe, or some unspecified entity or power.
This is the construction that we see in the phrase Fuck you. It is an invocation that the addressee be fucked by an unspecified somebody. It is not an imperative. Notice while we're here that the illocutionary force of some of the expressions above is that the speaker is very surprised or shocked about something.
Now the Original Poster feels that fuck me is an imperative construction. While this may be true if you find it in the dialogue of a book such as D. H. Lawrence's Lady Chatterley's lover, this isn't the case when these words are being used with their idiomatic non-literal meaning.
We noted above that some of these "subjunctive" idioms we've been talking about are used to express profound surprise. One of the pertinent examples being:
This particular idiom has many variations such as well knock me down with a feather!. The idiomatic Well fuck me! is not an invitation to the listener. Rather, it has the same type of literal meaning as fuck you but with a different direct object. However, rather than being used to express distaste at oneself, it is, just like knock me down with a feather, an expression of awed surprise.
You can read about this type of usage of the subjunctive in A Communicative Grammar of English by the eminent English linguists Geoffrey Leech and Jan Svartvik (co-authors of the Comprehensive Grammar of the English Language). You can get a snapshot from Google books here.
As Leech and Svartvik don't explicitly mention the more colourful examples dicussed by the Original Poster, you can read a few lines about them in Davidsen-nielsen's Tense and Mood in English, although you should be aware that this is really a comparative grammar of English and Danish.
"Fuck you" is probably best described as an imprecation or a curse, which is usually a wish that something bad happen to the person you are cursing at.
Look at #2 under the entry for 'Fuck' in the OED:
Used profanely in imprecations and exclamations as the coarsest equivalent of >DAMN. v. 5
1922 JOYCE Ulysses 587 God fuck old Bennett! 1929 F. MANNING Middle Parts of >>Fortune II. xv. 379 'Fuck the bloody thing!' he said fiercely under his breath. >>1955 S. BECKETT Molloy 69 Fuck the son of a bitch. 1959 F. KING So Hurt / >>Humiliated 151 'Suppose any of the neighbors were to look out and see them.' >>'Oh, f--- the neighbours!' 'Really, Henry!' 1969 'J. MORRIS' Fever Grass ii. 24 >>Why don't you ... tell whoever it is to go fuck themselves?
A example of a milder form (in English) is damn you, or God damn you.
'Getting fucked' is usually associated with being penetrated, and the implication in the imprecation is that it would be shameful and not pleasurable for the person getting fucked. See also Catullus 16 for a classical precedent.
Literally, it's the vulgar way to say "get sodomized", by somebody else's penis or, as Hot Licks suggested, by having someone insert some object into your rectum. But don't take it literally. What you really want is to express your anger and sound offensive. I suggest you don't use it unless you have a strong reason. It's comparable to "go to hell"; surely the person won't follow your advice and you don't expect them to.
The English language, as evidenced in usage by the dominant cultural standards of the British Isles and the United States, has a very peculiar relationship with the word "fuck". And from now on I'll use "frack" (from the Battlestar Galactica series) just to try to avoid being too offensive to some people.
As eatshaw has remarked, "frack you" is an imprecation. For the last century or so it has been considered the ultimate in offensive obscenity, and has taken on quite a number of different uses and meanings. Obviously, "frack" means to have sex. If you consider this, the use of the term as an insult seems a bit odd. An awful lot of people want to have sex, so how is "get fracked" an insult? Is "Make a lot of money!" an insult? How about "Live a long and healthy life"? "Be happy!"? You see the problem.
Regardless, many uses have only a rather vague meaning, since the base term is (or was, depending on culture) so overwhelmingly offensive. "Frack me" is generally used as a term of exasperation or frustration, or even weary resignation, while "Well, frack me sideways" implies overwhelming astonishment. Of course, if a woman looks you in the eye and says, "Frack me! Right now!" it's a clear and explicit invitation to take the word literally.
"Frack you" has been described as the equivalent suggesting that the object of the speaker's displeasure ought to be sodomized, but it's important to keep in mind the qualification "but don't take it literally". Rather, the offensiveness of the term is being turned on the recipient, and there is no need to conjure up detailed explanations. Although, it's true, the shadow of the meaning is always there. If a male is told "Frack you!" and is either very secure in his heterosexuality or very comfortable with his homosexuality, he can respond with something like "Don't you wish" or "In your dreams".