Rancorous applause is actually pretty good, in my opinion. I personally like it more than the other options you listed. It also happens to sound good (nice meter to it), which is important in certain types of writing such as fiction or magazine pieces. So I'm not sure what you mean by "natural"? If you mean more ordinary, then hateful might work fine for you. I wonder if it's really possible to have resentful applause -- I'm having trouble picturing it.
Looking at a thesaurus page for rancor (http://www.thesaurus.com/browse/rancor), there are a couple of possibilities that stick out for me:
Ultimately, if the meaning is extremely important, you should choose the synonym that is the most accurate. (Take the time to look up the words in the dictionary.) I personally think rancorous applause and venomous applause are colorful, poignant phrases that will make your writing memorable to the reader.