In the sentence provided as example, "abused" is the best option.
abuse (verb) = to use something for the wrong purpose in a way that is harmful or morally wrong:
- She is continually abusing her position/authority by getting other people to do things for her.
- I never expected that he would abuse the trust I placed in him.
to abuse alcohol
And the sentence becomes:
The Frostbite engine was used to make a RPG, because it was abused it led to several problems during the development cycle.
However, this implies something that should be (normally) done.
In some cases, the use of the "not optimal tool" is actually the better thing to do, when you have nothing better at hand:
He did not have a hammer, so he worked it around using a dictionary.
based on the word "workaround":
workaround (noun) = a way of dealing with a problem or making something work despite the problem, without completely solving it:
- The software still has a few glitches but we've come up with a workaround.
Of course, the limit between and "abuse" and "workaround" can be quite blurry sometimes, "proven" in the sentence below:
He did not have a hammer, so he worked it around by abusing a dictionary.