We had a chat (while) drinking a cup of coffee.
This describes two activities taking place at the same time.
We had a chat over a cup of coffee.
This expresses the coffee as the context in which the chat took place. You might think of the coffee as the foundation ("underneath") of the chat, hence, the chat is described as being "over" the coffee.