In common English usage, it is not considered to be as offensive to the recipient. However, based on its history, it should probably be considered more offensive than it is, and both white and black Americans may be offended. It should be avoided if at all possible.
The term "white nigger" came into common use in American English in the 1830s, and the similar term "white trash" came into common use a little later, as a more polite form of essentially the same concept. ("White nigger" had somewhat different meanings and implications in early UK usage that I won't get into).
The idea behind calling someone "white trash" specifically is to describe them as someone who is white, but acts trashy. The clear implication is that what is unusual is not that the person is trashy, but that they're a white person who is trashy--or, in other words, that non-whites are expected to act that way, but white people ought to know better. Acting in a "less than white" way gets you labeled as a "white nigger" or "white trash."
In this sense, it's an inherently racist insult. It was used in the American South in the nineteenth and twentieth centuries by upper-class whites to describe lower-class whites, who the upper-class whites saw as equivalent to black people--and those southern elites didn't think much of black people, as you may recall.
So "white trash" is offensive for the same reason it's sometimes considered offensive to describe a black man as "articulate" or to say that someone is pretty smart "for a [fill in the blank]"--because its usage implies the derogatory stereotype it's meant to be an exception to.
That said: not everyone makes this connection, and many white and black Americans use the term as a synonym for cracker, to describe a specific social stereotype of a poor, rural, uneducated, working-class, evangelical Christian white person, often from the American South. In modern usage it is considered offensive to people who identify themselves as working-class and rural, but not to all white people. The less racially charged term "trailer trash" is sometimes used instead.
Depending on the person you are talking to, "white trash" will probably be considered fairly offensive--more so than other cultural referents like "soccer mom" or "valley girl," but not as offensive as the most offensive group slurs, like "nigger," "faggot," or "kike."
It would probably be placed on a par with insults like "Oreo" or "Uncle Tom," terms that are divisive enough that you would never use them in a formal setting and that they might provoke a fight if you used them to someone's face, but that are not considered entirely taboo.
Edit: Because I was asked in the comments, I'm adding a quote showing the use of "white trash" and "white nigger" interchangeably.
From a first-person account of growing up in the Jim Crow south:
What anybody will tell you, though, to prove that he or she is not prejudiced, is that the worst kind of nigger is a white nigger, which is defined to be a white person who has become completely niggrish, who lives even worse than a nigger. White trash might as well be niggras, have all the bad habits, smell just as bad, cook dirty food, live in filthy houses, and thrive on squalor.
I'll add more if I can find them.