You can use "switch" or "exchange", but the verb "swap" is specifically targeted for this kind of situation:
I swapped my red scarf for her blue scarf.
As James K points out, "swap" automatically implies mutual consent, while a "switch" or an "exchange" could be done surreptitiously.
Note I use "swap for" not "swap with". This is because you also "swap with" a person, "for" some other object:
At the beginning of each month all of us roommates have a random drawing to determine chores. I hate washing windows, so I swapped that with Fred for mowing the lawn.
It's not wrong to say "swap with", but in my mind this focuses the action on the objects swapped, rather than the exchange with another person.
Right before going to the cashier, I swapped the red dress with the blue dress.