"They told her over and over again that she must not do it." is a correct sentence. Is "should" OK instead of using "must" in this sentence? If so, "They told her over and over again that she not do it." might be OK, too?
They told her over and over again that she should not do it is OK.
They told her over and over again not to do it is also OK.
Those two sentences and your original sentence (with must not) are all correct, but they do not mean the same thing.
(They told her over and over again that she not do it is not grammatically correct, however.)
The words "must" and "should" are often interchangeable, but at the same time, there is a clear difference in meaning.
You must (or, more commonly, have to) do x means there is some external reason to do x: there is a rule, or a logical requirement.
- To pass the course, you must complete the final project.
- I have to buy some flour this afternoon, because I'm baking bread tonight.
- You have to obey the safety regulations, or you'll be arrested.
- We can have this conversation later, you have to go now, or you'll miss the talk.
- You have to pay your taxes.
You should (or, sometimes, ought to) do x means that it is of a benefit to you, or, that it is the morally correct thing to do.
- To do well in the course, you should study every night.
- I should buy some flour this afternoon, in case anyone wants to bake bread later.
- You should obey the safety regulations, or you might hurt yourself or someone else.
- You ought to go, you look very tired!
- You should give money to charity.
Less precisely, we can think of "should" as a "lighter" version of "have to," which is why both of them work in many sentences (with only a very subtle distinction in meaning). However, there are definitely sentences where we can contrast them.
You don't have to attend the lecture, but you should.
This suggests that there is no rule or other sort of obligation on the listener to go to the lecture, but that the listener will nonetheless benefit from going (or perhaps, that going is the morally correct thing to do).
Back to your sentence.
They told her over and over again that she (must not)/(should not) do it.
With "must," you suggest that there is a (formal or informal) policy in place, and that it would be a violation of the policy to do this thing. (Or at least that "they" think this.)
With "should," you suggest that it would be a strategic mistake (or ethically wrong) for her to do this thing. (Or at least that "they" think this.)
There are many contexts in which your sentence could appear where both words would be acceptable, and where the meaning would be only subtly different.
Your third sentence, with neither "must" nor "should," does not work at all after "told". You might be thinking of something else; namely, it works with some other verbs, "asked" or "demanded," but is a bit archaic sounding. (It is the subjunctive.)