These sentences are using a strict transitive sense of the verb to operate, where the subject is the person doing the operating and the object is the machine being operated. If you're sticking to this transitive sense of to operate, then no, you can't correctly say that the machine is operating, because that puts the machine as the subject, not the object.
However, there is an intransitive definition of the verb to operate that is a synonym of to function. You can say that a machine is operating in the same way that you can say that a machine is functioning. You probably would not use this intransitive sense in your second example:
Do Not Talk To Operator While Machine Is Operating. [This is awkward.]
The reason it's awkward is that it first implies the transitive operate by introducing an operator (that is, someone who operates the machine), but then immediately switches to the intransitive operate by using machine as a subject within the final phrase. This poses a bit of a logical dissonance by using two different definitions of the same family of words within the same sentence.
You could do the substitution in the first sentence without much change in meaning:
Do Not Open While Machine Is Operating. [Not awkward]
Here there's no confusion between the transitive and intransitive operate since there's no sign of the transitive form at all. However, you have very slightly changed the shade of meaning by implying that the machine itself is operating independently of any operator, whereas the the original form is more consistent with a machine being operated by a human. Another alternate to the original version is to use the passive voice more explicitly:
Do Not Open While Machine Is Being Operated.
This last version very strongly suggests that the machine is directly under the control of a human operator, in contrast to the "is operating" version.