Would you say
My family used to be poor but not anymore. Normally, you would say I am from this country because that is your native country. However for this question your origin has changed from being poor to not being one. I would use came/was from a poor family. But I heard someone say "I am from a middle class family" even though that person is rich now.
I come from a poor family is a more idiomatic way of speaking than I am from a poor family although both are grammatically correct.
I am from a poor family sounds slightly more poetic or theatrical.
To say: I came from a poor family implies either that you are telling a story or that the family is now better off.
If the latter, it would be better to say clearly that your family was poor at the time that you were born.
It would be natural for a writer to use the past tense in a biography or novel that is written in this tense.
Regarding the statement you heard, it's perfectly fair to say that one comes/is from a middle class family even if the speaker has subsequently grown wealthy. It implies that the (rest of the) family is still middle class, whether financially or in its outlook and habits.
Rather than saying "I came/come from a poor family, consider:
I grew up in a poor family. or even I grew up poor.
This keeps the focus on you and your life when you were young, and doesn't get into what may be a side issue of what may have happened to your family later. (What happened later may be part of the story, but you can't get everything into one sentence.) You could also say:
My family was poor while I was growing up. This actually suggests that they may no longer be poor, which may be what you want.
