This is an example of the semantic number, and the grammatical number not being the same
You may use "it" and "was", to agree with the grammatically singular "Khmer Rouge", or you may use "they" and "were" to agree with the semantically plural "Khmer Rouge". Both are correct.
For example, from TIME:
Simultaneously, the Khmer Rouge were planning the steps necessary for a radical shift to an agrarian society
The Khmer Rouge was the strongest partner in this coalition, which carried on guerrilla warfare until 1991.
In American English it is more common to follow the grammatical number (Americans would be more likely to say "The government is...") British speakers are more likely to follow the semantic number (Britons are more likely to say "the government are"). But this is not a hard rule, and there is a lot of variation on both sides of the Atlantic.
It can be used to add nuance "The Khmer Rouge was..." suggests that it was a single body that acted like a unit, rather than an association of different people.