To where did you go yesterday?
Where did you go to yesterday?
Where did you go yesterday?
I see people omit the 'to' a lot. However, I also asked a related question yesterday
Who did you give the keys to?
To whom did you give the keys?
Here, the 'to' seems necessary. Why can you omit the 'to' when talking about places?