Any of the animals I can think of that you might actually use for travel in real life are basically of the horse variety (horse, camel, mule, donkey, etc.). Maybe throw elephants in there too.
For any of those, I think either
- "by [animal]," or
- "on a(n) [animal]"
would probably work.
You would not say travel "on donkey," though. That doesn't sound idiomatic, so it could leave a native speaker trying to come up with a weird interpretation that you didn't intend.
In the case of some other more exotic animal, while "travel by" would be grammatically correct, it's not likely to be used. For example, if I had some crazy friend that rides a Komodo dragon to school, I probably wouldn't say that my friend goes to school "by Komodo dragon" because that makes it sound too mundane, as if lots of people go to school on Komodo dragons.
However, you might read something about people traveling, for example, "by dragon" in some book where the author wants to make it sound like that's a normal occurrence in the book's universe.
As for traveling by/on "animal," that also does not sound particularly idiomatic either, though it could be correct in a certain context. Generally, just saying "animal" is a bit too vague. Since there are relatively few animals you can actually use for traveling, this phrasing might leave a reader/listener imagining that some a-typical animals are being used for travel. (The reader/listener may think that since you didn't just go ahead and say, for example, "by horse or donkey" that you must be implying that some other a-typical animals are used for travel by whomever you're talking about.)
However, if you had already introduced the ideas that a certain group of people use X, Y, and Z animals for traveling, you might get away with saying "by animal" as a shorthand for "by X, Y, and Z" later on in the text.
The other time I can think of that "by animal" might work would be if you actually did want to imply that something other than the typical group of horse-type animals were used for travel. Say, maybe you're writing a text about ancient humans who traveled around on some animal that is now extinct. You don't want to say "by horse" because it's incorrect, but you also don't want to give the scientific name of the extinct animal because it sounds clunky, and there's no colloquial name for the animal because it's extinct. So maybe then you could just say "by animal."