Do they both roughly mean the same thing? When I am thinking of the phrase, I think of someone shooting someone's brain and blowing it to pieces? Are both of these phrases used to say such a thing?
There is a small difference between the two.
Blow the brain up only says that the brain will be destroyed (to blow up meaning to explode).
Blow the brain out indicates that the brain will leave the skull, intact or otherwise.
It can be assumed that most events that cause the brain to explode will probably rupture the skull too, however "blow the brain out" is explicit in the brain leaving the skull.
Finally, blow [subject's] brain out is a well known phrase, so would more immediately convey the meaning.