Is it an erroneous phrase?
It's not common, that's for sure. I would say it's erroneous in that it's missing the word "get", or "become": "they felt themselves get quieter". But even then, that would leave a native speaker thinking about your choice of words rather than what you're saying.
when can one say "I feel myself" or "They feel themselves"?
The "feel" in this sentence structure is usually referring to a physical feeling. If you just say "I feel myself", that usually means that you are touching your own body. Lacking context, that can easily be understood sexually.
If you say "I feel myself [something]", that something would be something physical, but involuntary - not something you are intentionally doing. For example, "I felt myself fall asleep."
Or even something that you are "intentionally" doing, but feel powerless to stop. For example, an alcoholic saying "I felt myself reaching for another drink."
Or you could say something like "I feel myself getting angry". That would be understood to mean that you have noticed the physical responses that go along with being angry before you realized that you are actually angry.
And that is the problem with "they felt themselves quieter immediately" - "quieter" is not a physical feeling, it's auditory. A native speaker would more likely say:
- "They heard themselves get quieter", or
- "They noticed themselves get quieter"
I'm a native English speaker from Canada.