What does it mean when we say someone is "not a man of comforts" ? Does this mean he's not a rich man?
That's one meaning. "Man of comforts" is a rather old-fashioned expression, meaning a person who is accustomed to having the "finer things in life" about himself, so by extension well-off to the point of being wealthy. So, a person who is "not a man of comforts" is either a well-off person of simple tastes, or a poor person.
A little research yields examples that illustrate the different uses (first, the poor man):
and (the well-off man)
and (the well-off man with simple tastes)
It mostly means the speaker/writer is seriously out of touch with contemporary English usage. There are only 7 instances of "a man of comforts" in Google Books, so it's obviously difficult to say the words have any inherent special meaning beyond the literal interpretation.
In fact, the earliest (1660) of those 7 written instances apparently does refer directly to comforts as elements of cheer in an otherwise woeful life. And it would be perfectly possible for someone of modest means to say...
Of course, a richer man might have more comfortable "comforts of life" than the "poor, but content" man. But the above usage isn't uncommon, so it's not unreasonable to interpret OP's phrase as meaning a person who both has and appreciates the comforts of life - indicating an attitude to "comfortable things" rather than possession of the means with which to buy the best ones.
But most likely if you do come across the expression used today, it'll be a "pseudo-archaic" allusion to...
...which is a "current expression".