The shortest form, (3) is the correct one.
When I mix liquid A and liquid B the new compound turns green.
The second variant ("turns to be green") has superfluous "to be" in it. One of the meanings of the verb "turn" is "to change into", especially when talking about colors.
The first variant ("turns out to be green") is incorrect because the meaning of "turns out" has a delayed perception or discovery in it. It would be more appropriate to say, for instance,
When I mix liquid A and liquid B the new compound turns out to be highly acidic.
because it is not apparent from the mixture what level of pH it has and additional tests (to discover the acidity) need to be performed. Those results can [later] lead to finding out that the "new compound" is highly acidic.