What does it mean when we say someone is "not a man of comforts" ? Does this mean he's not a rich man?
That's one meaning. "Man of comforts" is a rather old-fashioned expression, meaning a person who is accustomed to having the "finer things in life" about himself, so by extension well-off to the point of being wealthy. So, a person who is "not a man of comforts" is either a well-off person of simple tastes, or a poor person.
A little research yields examples that illustrate the different uses (first, the poor man):
One day, before the city
Of Brussels, in Brabant,
We saw with fear and pity
This man of comforts scant,
– The Ladies' Garland and Family Magazine, 1846
and (the well-off man)
Tiernan, it proved, was a man of comforts, holding a well-padded county job.
– Mounted Justice: True Stories of the Pennsylvania State Police, 1922
and (the well-off man with simple tastes)
Robert Pattinson of 'Twilight' not a Man of Comforts
It mostly means the speaker/writer is seriously out of touch with contemporary English usage. There are only 7 instances of "a man of comforts" in Google Books, so it's obviously difficult to say the words have any inherent special meaning beyond the literal interpretation.
In fact, the earliest (1660) of those 7 written instances apparently does refer directly to comforts as elements of cheer in an otherwise woeful life. And it would be perfectly possible for someone of modest means to say...
"Why would I want to win the Lottery? I have all the comforts of life in my own little home already"
Of course, a richer man might have more comfortable "comforts of life" than the "poor, but content" man. But the above usage isn't uncommon, so it's not unreasonable to interpret OP's phrase as meaning a person who both has and appreciates the comforts of life - indicating an attitude to "comfortable things" rather than possession of the means with which to buy the best ones.
But most likely if you do come across the expression used today, it'll be a "pseudo-archaic" allusion to...
comfortably off - rich enough to pay for everything you need
...which is a "current expression".