I remember being taken to the zoo.
This is correct and sounds very normal to native ears. Being taken to the zoo functions as a noun phrase denoting the act in which you were taken to the zoo. Being is a gerund here. The passive construction here omits the subject; the implied subject is me (which a person would only say in order to create unusually strong emphasis).
It might help to look at some comparable sentences where the subject of being taken is explicit:
I remember John being taken to the zoo.
I remember him being taken to the zoo.
Notice that the subject of being taken is in the objective case. That's because it's also the object of remember, the main verb of the sentence.
Sorry, I have to tell you this
There's also an older school of thought that says the previous two sentences are incorrect and should instead be:
I remember John's being taken to the zoo.
I remember his being taken to the zoo.
In this parsing, the object of remember is being taken to the zoo. The subject of the gerund in this construction takes the possessive case (strangely enough).
Most fluent speakers today hear both him being taken and his being taken as correct. That is, people can parse both forms. The first form works by analogy with I helped him learn and the second form works by analogy with I helped his education.
They are remembered taking me to the zoo.
This is actually correct, just a little unusual. Here, taking is a present participle, not a gerund. Taking me to the zoo modifies they, in the manner of a subject-complement. A comparable sentence might make this clearer:
Football players are usually drawn running with the ball.
In other words, in most pictures of football players, the football player is running with the ball.
The reason your example sentence is unusual isn't because of the grammar, it's because it's a little hard to imagine a situation where people would be remembered that way. But it's certainly possible and the sentence can be understood. Perhaps someone took a photograph of "them" while they were taking you to the zoo, they died long ago, and since then, many people have seen this photograph.