They all mean slightly different things.
She won't open the door.
There is no ambiguity here - she will not open the door, it will not happen, she will not do it. It passes no comment on the reason why not.
She is not willing to open the door.
This still means that the door will for certainty not be opened, but "unwilling" means "not ready, eager, or prepared to do something", so this is getting into the person's motivation. Perhaps she might open the door if motivated to do so?
Probably she may not open the door.
This isn't quite right - "may not" expresses possibility, so the opening word "probably" is redundant. I think this should either be:
- "She probably won't open the door"
- "She may not open the door".
With different stress though, the latter could also mean that she can open it, as a secondary meaning of "may" is that someone is permitted to do something. Otherwise, both of these mean that there is only a possibility she will open the door.