In some contexts, there is little difference between the phrases in your question title.
However, they do have different uses and meanings—and other contexts make that clearer.
That being so.
Here, you are effectively saying, "given that" or "because that is true."
You will follow this up with something else that can now occur because of the previously stated fact:
Gravity causes unsupported items to fall downward. That being so, when I drop this pen it falls to the floor.
Using a magnifying glass to focus sunlight can intensify its normal effects. That being so, it can be used to burn things.
With that said.
I normally encounter this phrase as something that follows an obligatory statement and precedes what you really want to say. Or, it can be used as a "disclaimer" for what is to come.
First, I need to thank the Academy. With that said, I'd like to talk a bit about my life and how I got to this point.
English is not my native language. With that said, here's what I think of the sentence.
Having said that.
(I am changing this to that because it sounds more natural to me. But you can continue to think of it as this if you wish.)
This is normally encountered as a way of almost contradicting what you have just said and continuing on with something that wouldn't normally follow. You can think of it as "however," "in spite of that," or "despite that."
The path to the left will get me to my destination faster. Having said that, I am going to take the path to the right because it's more scenic.
I know that I should not be eating dessert instead of dinner. Having said that, I am going to order pie.
If you take my example sentences and replace one phrase with another other, you should see that some no longer make sense at all, while others sound slightly less natural.
Using a magnifying glass to focus sunlight can intensify its normal effects. Having said that, it can be used to burn things.
It sounds a little strange for the second sentence to follow the first in this way.
I know that I should not be eating dessert instead of dinner. That being so, I am going to order pie.
This sounds very strange, and the second sentence doesn't follow the first at all.
Note that with that said can often be replaced with having said that, but the reverse is not true.
First, I need to thank the Academy. Having said that, I'd like to talk a bit about my life and how I got to this point.
While okay, it's a bit less natural than the original.
The path to the left will get me to my destination faster. With that said, I am going to take the path to the right because it's more scenic.
This makes much less sense.