Being annoyed is different from being angry. If I am angry at someone, I might yell at them. If I am annoyed, I might just roll my eyes and shake my head. If I am greatly annoyed, I might exaggerate the eye-rolling and head-shaking, but it's still not the same as being angry.
Naturally this depends on context and personality. There are people who yell even when they are only annoyed. There are people who don't yell even when they are furious.
In addition, "annoyed" can be a diplomatic substitute for "anger". In a professional environment it is often inappropriate to be or act angry, so instead we use "annoyed" (or "upset") to downplay the severity of the emotion.
HR Rep: Ted, we've called this meeting because we were told you got a little annoyed with your boss the other day.
Ted: Yes, I was a little upset.
HR Rep: Well, your coworkers said you were screaming and running around the office, yelling out some very rude words to describe your boss.
Ted: OK, I guess I was more than a little upset.
For this reason we can only guess what "great annoyance" actually means. More context is needed to understand what actions resulted from the annoyance of his colleagues. Only then we can tell whether the author is downplaying a more serious emotion (like anger), or if it means the colleagues simply rolled their eyes and shook their heads frequently and intensely.