I'm not sure if I have heard both phrases used, but something in my head tells me I have, and I am just wondering if they are both correct (I would appreciate it if you could provide an explanation). Another related question: is it possible to say "error of/in translation"? This question has been bothering me for quite a while. Thank you for your answers.
Both "an error of ..." and "an error in spelling" are reasonable. I would use the second when talking about a particular example
He made an error in spelling the word 'necessary' with two 'c's.
I would use the first when not referring to a specific example.
His essay contained five errors of spelling.
But I'd be more likely to say:
He made five spelling errors.