"Insides" refers collectively to all of one's internal organs. The singular is not used in this case; if you're only referring to one, you can specify which organ.
"Gnawing on my insides" is an expression, so trying to make it into literally "gnawing at what's inside of me" ruins the colloquialism and makes it sound like maybe you're being eaten by rats.
The other side of the expression would be
the anxiety that had been gnawing inside me for so long...
but notice this use of the singular inside does not refer to one's entrails, but to the location of the feelings, and says nothing of ones guts.
So "insides" means guts, but "inside" only refers to where your insides are not the insides themselves.
EDIT:
Is it correct to say: "The gnawing in his inside(s)", or "The gnawing in him" ?
Both of these would depend on context. I would write them as:
"The frustration gnawing in his insides", or "The anger gnawing in him"
in order to make it clear you are using the figurative "insides" (plural when referring to innards), but if the context make this clear, then the way you wrote them would be understandable as well.
Another similar expression is:
The worry eating away at me...