"If that" is a phrase used when someone has made a comparison (generally a comparison to something poor). "If that" means that it may be even worse than the comparison implies.
A: The company will only just meet its sales targets this year.
B: If that.
B means that the company might fall a good way short of the sales targets.
In the example "If that" could refer to two previous statements. It may refer to "This is a bad idea", and would mean that the idea is 'worse than bad'.
The second is that it refers to "This conversation is between you and me", and the meaning would be "it isn't even between you and me", presumably because Chandler didn't understand it or because he didn't say anything. In either case it's a conversation Joey is essentially having with himself.