This is a very modern and technical term, so there isn't much of an historical basis, and probably not rigid usage. The differences you are trying to highlight are contextual, and general definitions are likely to vary depending on the focus or application. If there is a nuance in the choice of terms, though, I would suggest this:
"Message" focuses on the individual communications. "Messaging" focuses on the system that enables the messages.
A "message system" might focus more on an application that handles your individual messages, for example, organizing and displaying them, composing and sending them. A "messaging system" would focus more on the infrastructure that supports it, for example, the application itself, the protocols used, the transmission mechanisms, input mechanisms, etc.
Protocol would be even more subtle. As Cantalouping points out in a comment, there isn't really a practical difference. Protocols, in general, cover a range of things, including the structure of the message, how the message is transmitted, etc. So if there is a nuance, it would be of the nature of word association that some people may have, more than a difference in actual definition.
The "message protocol" might conjure an assumption that you are referring to just the structure of the message (things like header characters incorporated into the message). "Messaging protocol" would always be interpreted as including all of the standards relating to the message and transferring it.