Words are :- theater, behavior , litre, dialogue , tire, program, omelette, cheque, pajamas, realize .
I think you're looking for a hard and fast rule, and there isn't any. We can make some limited use rules:
- -er (US) vs -re (UK): theater, center, liter
- -or (US) vs -our (UK): behavior, savior, color
But there's a lot of cases that are one-off examples, and these rules don't apply 100% of the time. Standardized English spelling is relatively new, historically speaking. So tire/tyre are clearly pronounced the same, just spelled differently. Similar for realize/realise. For your particular examples:
- Typically US spellings are theater, behavior, liter, program, check, realize, pajamas
- Typically UK ones are theatre, behaviour, litre, programme, cheque, realise, pyjamas (at least, I think so, not having spent a lot of time reading modern British publications)
Some of these aren't country specific, though. "Dialogue" is typical on both sides (until being simplified mostly for computer use as "dialog"), as is "omelette." "Dialog" is a great example of how words change over time regardless of where you are. And, of course, sometimes people use the alternate spellings to show allegiance to their home country, to be different, for dramatic effect, for a dash of humour, etc. Despite the fact that I am an American, I will often spell theater/liter as theatre/litre, not for really any good reason.