To conveys movement towards a destination.
You would say:
I went on a trip to the zoo.
Because what is important is the trip, which implies movement. Saying:
I went on a trip at the zoo.
Means that you were at the zoo, and then went on a trip to somewhere else when you were there.
This applies to all movement-related verbs:
I travelled to New York this summer.
I went on a journey to America last year.
I drove to work this morning.
Long story short: with "to" you move towards the destination, with "at" you act while being there.