Lambie's answer correctly presents the prescriptive rule that's taught in grade schools, and with your sentence, the prescriptive rule happens to agree with how native speakers would interpret that sentence out of context; but it's important to be aware that in real English, there's more flexibility than that answer claims. For example, it's not hard to find examples like this one:
Presently a man came by who was a stranger to that neighborhood and […] [link]
where the relative clause "who was a stranger […]" modifies the subject "a man", even though the predicate "came by" appears between them, or like this one:
[…] she lowers her standards and lets a man into her life who is beneath who she is, […] [link]
where the relative clause "who is beneath […]" modifies the direct object "a man", even though it immediately follows the noun phrase "her life".
I don't think most readers would bat an eye at sentences like these; they sound perfectly natural, and make perfect sense.
But these sentences only work because it's very obvious which noun phrase the relative clause is modifying: there's only one candidate that's even remotely plausible. With your sentence, by contrast, unless the context gives a very compelling reason to understand the relative clause as modifying "the man" rather than "the woman", I feel confident in saying that all readers would understand it as modifying "the woman". In fact, even if the context does give a compelling reason to understand it as modifying "the man", I think there's a good chance that readers would understand it as modifying "the woman" anyway, and would just be confused.