If you look into meanings of that (and there're many) in Merriam-Webster, one of the meanings is:
: the one : the kind
The richest ore is that found higher up.
In this example, there's only one richest ore, and the phrase following that explains where it can be found.
Similarly, in your sentence:
The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.
There's only one kind of problem (The fundamental problem) and that is used to refer to it.