[I'm compelled to write an answer here because the accepted one sounds good and is highly voted but is wrong.]
Preliminaries
The word that is not a pronoun and doesn't stand in for a noun phrase. It is the same that which we find in sentences like:
- I think that he hit him.
The idea that we can transform sentences by replacing an it or other pronoun in the subordinate clause with a that will not go through. Consider this proposal from the other end:
- It is not to [you] that I spoke.
This does not decompose into:
- It is not to [you]. I spoke [you]. (the last sentence is ungrammatical)
Similarly, consider:
- This is not [the place] that we met.
This does not decompose into:
- This is not [the place]. We met [ it ]. (the last sentence means something different!)
How that relative clauses work.
Relative clauses with that, and those relative clauses that can occur without any relative word both work in the same way. The relative clause has an antecedent, which is often a noun phrase, but may be a different type of phrase, for example a preposition phrase as in (2). That example would be best represented like this if recast as two sentences:
- It wasn't [to you]. I spoke [to you].
Notice that to you is a preposition phrase, not a pronoun or noun phrase!
The important point about a relative clause with that is that is has a gap in it, and it is this gap that is co-indexed with the antecedent. In other words, the gap tells us where to mentally reinsert the antecedent into the subordinate relative clause. Consider:
- This is the man that [he hit]
The verb hit normally takes an object. The object is missing from the subordinate clause in (4). We interpret the sentence like this:
- This is the mani that [he hit ___i]
The little < i > there shows that the man and the gap in the relative clause refer to the same thing. So we understand from the gap in the subordinate clause in brackets that the man was hit. We could also represent the sentence like this:
- This is the man that [he hit
the man.
Compare (4) with (7) below:
- This is the man that [hit him].
Here the gap in the relative clause is in subject position. This sentence means:
- This is the mani that [___i hit him]
or
- This is the man that [
the man hit him]
We can also represent It wasn't to you that I spoke in a similar way, as long as we don't pretend that that is a pronoun:
- It wasn't to you that [I spoke].
- It wasn't to youi that [I spoke ___i
- It was to you that [I spoke
to you].
The reason that we cannot fill the gap in with words is that the gap actually tells us what role the antecedent plays within the relative clause. Consider examples (4) and (7) and then consider the ungrammatical example (13) below:
- This is the man that [he hit him].
We cannot understand from (13) whether the man referred to was hit or did the hitting because there is no gap in the relative clause to tell us where the man fits in.
The Original Poster's question
You cannot fill in the gap in a relative clause with a pronoun. It's ungrammatical to do so. The reason is that the gap is an important part of the relative clause. It tells us what the antecedent refers to.
The word that is not a pronoun, although traditional grammars used to think it was. All the evidence shows us that it isn't. A good way to see this is that even in relative clauses where we don't need to use the word that, we still cannot fill in the gap with a pronoun:
- This is the man we spoke to.
- *This is the man we spoke to him. (ungrammatical)
Here we clearly see that (15) is ungrammatical even though there is no word that. It is completely irrelevant whether the word that is present or absent, you cannot fill in the gap in a relative clause, with a pronoun or with any other phrase.