In all of your examples, "x to y" could also be "x over y", and this might be more common to hear.
I used to prefer milk to coffee in the past, but now I prefer coffee to milk.
Using both used to prefer and in the past feels redundant to me, as does the to milk at the end. I'd remove the bolded parts above.
I used to prefer milk to coffee 10 years ago, but now I prefer coffee.
This time, you can keep the 10 years ago, because it's adding information. Saying "I used to prefer" feels a little stilted in this sentence, I think "I preferred" works better. (also incorporating my "x over y" change I mentioned earlier)
I preferred milk over coffee 10 years ago, but now I prefer coffee.
I don't think I'd say that your original here was ungrammatical, but I would say it wasn't colloquial.
(user3169 answered the second sentence correctly already: it's not grammatical as-is because the words should be plural. Also, some of my same redundancy things apply. Being redundant can add emphasis or clarity in some cases, but I don't think that was needed here.)
(from your comment below)
do you think it is wrong to say "they invited me to the party yesterday but I preferred to stay home" because this, refusing to go go the party, is one time occasion (choice/preference). But I would not say " they invited me to the party yesterday but I used to prefer to stay home"
Probably not the first sentence, definitely not the second sentence. used to prefer shouldn't be used for a one-time occasion.
...or would it be more colloquial to say "I would rather have stayed home" / "I would prefer to have stayed home"
The first means you went to the party, even though you wanted to stay home (maybe you felt obliged to go), which doesn't seem to be what you were trying to say. The second could work.
Here are a few ways I might say this (the just is optional):
They invited me to the party yesterday but I just stayed home instead. or
They invited me to the party yesterday but I just wanted to stay home. or
They invited me to the party yesterday but I just wanted to stay home, so I did.
These all imply or explicitly say that I was invited, but declined because I wanted to stay at home rather than go to the party.
Sorry if it's hard for me to explain why some of your examples don't "feel right". I'm a native English speaker, and don't usually spend a lot of time thinking about why a certain sentence sounds odd, or how it should be said.