The claws scraped at a cloak they should have readily tattered, only to have instead one nail snap off. [Warcraft: War of the Ancients #1]
The structure looks strange and I didn't ever see "instead" in the position like this.
You are right to sense some strangeness here. There are two patterns being awkwardly combined.
Instead of tattering the cloak, one of the nails broke.
"Only to have..." is used when someone is disappointed or thwarted, so there should be a sentient being (a noun capable of having, i.e. experiencing) in the main clause:
He scraped with the claws, which should have readily tattered the cloak, only to have one of the nails break off.
The original sentence, in combining "instead" and "only to have", and in leaving out the sentient being, is something of a mess.
If the claws in this game are a sentient creature, that might explain the use of "only to have". Are they sentient?
P.S. OK, as it turns out,"the claws" is a creature. So we could say:
The Claws scraped at the cloak, which it should have readily tattered, only to have one of its nails snap off.
"Instead" could be placed at the end of that sentence (..."one of its nails snap off instead").
In your example, instead is used for contrast.
The expectation is that the claw would normally tear the cloak with little effort (scraped)
Instead can be used in several different places without loss of meaning
only to have instead
instead only to have
only instead to have
it has the contrasting meaning of but
but a nail fell off
the meaning being the nails on the claws were not as strong as expected
The (nails on the) claws should have easily ripped the cloak apart but they did not, and as a result (of trying to rip the cloak) one nail fell off
The general structure is
something was expected/supposed to happen instead something else happened
I was supposed to fly to London today, instead the flight was cancelled.
I paid for a coffee, instead I got tea.