You technically can do it, but it sounds awkward.
When you say "out of fear", you are only in part saying "because of." However, you are also saying that the obedience stems from it, that it's the originator.
You should use "because of" in cases like this when the thing in question is the catalyst for something to happen, and use "out of" when the thing in question metaphorically "gives rise" to something else.
However, it should be noted that this is highly metaphorical usage, and it's all about what sounds best. Let me show you how "because of" changes things:
They obeyed him because of fear.
This sounds like an explanation for why people in general obey. You can imagine an anthropologist saying this to explain why group X obey person Y.
Slightly more natural would be:
They obeyed him because of their fear of him.
Again, this sounds like a scientific explanation, rather than a simple explanation of emotional motivation. It almost sounds as if the fear is unfounded.
The others are clearer:
I asked out of curiosity. = Curiosity arose in me in this occasion and compelled me to ask.
I asked because of curiosity. = Curiosity is a thing which compels people to ask questions, and I have curiosity, ergo I asked a question.
I asked because of my curiosity. = I'm always curious, and so I ask questions all the time.
She did it out of spite. = She did not have any reason other than pure spite for doing it.
She did it because of spite. = Spite is a thing that makes people act maliciously, and therefore she acted maliciously because she had spite.
She did it because of her spite. (very awkward) = She has spite, so doing this sounds like a natural thing for her to do.