To be "up a creek" or "up a creek without a paddle" or "up s*** creek" is to be stuck in an undesirable place with no way out. This isn't necessarily the subject's fault but is generally considered to be a no-win situation.
To "paint oneself into a corner" is to have, by one's own actions/decisions, trapped oneself with no way out without ruining what's been done so far. With this idiom, the fault is laid upon the painter. It may come off as more of an embarrassment than a no-win solution, since escape is easy but requires re-doing everything—and getting paint all over you.
Going "down the rabbit hole" suggests going or being in an increasingly complex situation, with the suggestion that the time or effort remaining is unknown and prone to expand.
The OP asks for "the only way to get out is to go backward destroy everything you've done and start fresh ... because you have chosen the wrong method at the first step" which fits the "paint yourself into a corner" phrase perfectly. However, it doesn't have the finality of the OP's "you will never get out" requirement, though it does imply "you will never get out unscathed/(covered in paint)."
Personally, when I see someone start off wrong and come to a dead end—for example, in a logical debate—I'll use the "you've painted yourself into a corner" phrase.
When someone is in a bad situation they can't get out of, I'll use the "up (the/s***) creek without a paddle." (Or, if I'm being snarky and think they'll get the alluded idiom, I might say "I bet you wish you'd brought a paddle, huh?")