You belong to me I "own" you
- I have "rights" over you, I can make decisions on your behalf, etc.
Often with the implication that I
define the "value judgement" framework governing our relationship.
You belong with me
- It is right and proper that you should be close to me.
Often with the implication that the reason for this assertion is
either to comply with some "external" value system, or because it's in
your best interests (not necessarily just because it's what I want).
I have stolen this section completely from Fumble Fingers answer, as I agree with this totally. But I disagree with the analysis there.
Many people still romantically love the idea of "belonging to" someone, see this Dean Martin song:
You Belong to Me
Watch the sunrise on a tropic isle
See the pyramids along the Nile
Just remember darlin', all the while
You belong to me
Many people dislike the phrase, as they don't like being described as being owned by someone. This takes in the background of women literally being their husbands property for hundreds of years in the UK*.
To contrast here is the Police Song "Every Breath You Take", some people consider this a romantic song even though it was written to be a creepy song about a stalker citation of Sting
Every single day
Every word you say
Every game you play
Every night you stay
I'll be watching you
Oh can't you see
You belong to me
poor heart aches
With every step you take
And it takes in the
controlling nature of some relationships where one partner has the belief they are correct to control completely the other partner's every action and sometimes thoughts.
Even if people don't think of literal ownership, it sounds possessive and slightly controlling.
I would imagine the descendants of slaves are also not keen on this terminology, but I base this on nothing more than how I think I would feel.
So I would recommend not using the TO form yourself, as you never know how the other person will take it.
*I believe America is the same but my American social history is not good enough to make a definite statement.