Sam addressed the problem that was reported by Tom.
is probably the meaning you intend. Tom reported a problem, and Sam is talking about it.
Sam addressed the (reported by Tom) problem.
sounds convoluted, but could have some meaning.
If Tom reported many problems that were not really problems, and Sam was tired of hearing his complaints, then the problem is Tom. So Sam addressed the problem of too many annoying things reported by Tom. So:
Sam addressed the (annoying things reported by Tom) problem.
But this is a stretch (trying really hard to make any sense) at best.
the problemis your direct object so you need to put it after your verb, and
repirted by Tomgives more information about the problem so it should come after
Reported by Tomis called relative/adjective clause and cannot become before the word or phrase it explains.