"he didn't remember, and neither did I." Why does it become backward? "neither did I" but "not neither I did?"
Neither is one of a list of elements that, when first in a clause or sentence, compulsorily trigger inversion of the verb and subject. They don't have to come first, but if they do, the verb follows directly, then the subject.
They nearly all have some sort of negative sense.
Others are nor, never, nowhere, seldom, and rarely (but not occasionally, which has the same objective meaning as seldom and rarely, but without negative connotation).
In the same category as rarely, I note scarcely and hardly, but only in the sense of "the action had almost not finished", eg "Scarcely had I arrived home when the phone rang". In other senses, thsey cannot stand first or trigger inversion.