It's a quirk relating to anatomy in certain contexts. The key lies in the fact that the tiger itself is the object of the sentence, not its tail.
When we describe actions done to a person, we may describe the action being directly imparted to a specific part of their body, but we usually don't. We say it is being done to the person, then modify it with a special prepositional phrase, using "the" instead of a possessive, to describe what part of their body it happened to. This has the advantage of eliminating any ambiguity as to whose body part is being referred to.
I caught a tiger by the tail. (I caught a tiger. The part of its body I trapped happened to be its tail.)
Jim hit the tax collector on the head with a frying pan. ("On his head" probably means the same thing, but maybe a tiny tax collector is standing on Jim's head?)
"Come on," I shouted, dragging him along by the hand.
She lifted the pot by the handles.
I kissed her on the lips.
You'll have noticed that we extend this to animals and objects, not just people. So where does it stop? For one thing, we almost always revert to possessives if the part of anatomy must be described using more than one word:
She dragged him along by his left hand.
I lifted the pot by its ornate brass handles.
Don't confuse this with relative clauses, though:
We hung the ornament by the string that protruded from its top.
Second, if we do want to emphasize that the action is happening only to one part, not to the person or thing as a whole, it's usually best to say so directly. This often changes the meaning, though it doesn't always.
I caught a tiger's tail. (The rest of the tiger is long gone.)
She grabbed his hand. (Roughly the same meaning as "grabbing him by the hand," but less illustrative of what happens next.)
One last notable exception: "Feel sick to [one's] stomach" is an idiom all its own.
With all this said, though, "catch a tiger by its tail" doesn't sound all that weird. I can't think of any context where it would be the clear best choice, but neither does it particularly stand out. Don't worry about any of this inordinately.