I was out with a friend. A stranger heard the way we talked with each other. The stranger became offended at some of the vulgar language we used and I found him a bit uneasy and uncomfortable; so I decided to tell him in English:
Don't worry, he and I have jokes together.
Connotation: we often joke with each other.
I was wondering if the bold part above sounds correct to you. If not, then what is the naturally idiomatic way to express this matter?