The founder of Georgia, James Edward Oglethorpe, served as chairman of a committee charged with investigating prison conditions, which led him to take a special interest in the plight of debaters.
Which refers to the fact that he served as a chairman of a committee(charged with investigating prison conditions).
In other words, he served as the chairman of a committee charged with investigation prison conditions, and it was during this period when/that he developed a special interest in the plight of debates.
"James Edward Oglethrope" is a non-restrictive clause, it provides more detail but could be removed and placed in a separate sentence. When a few words are separated by commas a correct sentence will still be correct with the words removed. I have separated the non restrictive clause from the main sentence below.
"The founder of Georgia was James Edward Oglethorpe. He served as chairman of a committee charged with investigating prison conditions which led him to take a special interest in the plight of debaters"
There are still several chunks that don't change the meaning of "which" in this sentence. I have highlighted these chunks below.
"He served as chairman [of a committee] [charged with investigating prison conditions] which led him to take [a special interest in] [the plight of] debaters"
Once the bold words are removed we are left with a short sentence.
He served as chairman which led him to take an interest in the debaters.
He [did something] which [caused something].
Now we can see that the usage of which refers to his service as chairman.
In OP's example, which introduces a non-restrictive clause (containing parenthetical, optional, additional text, as opposed to a restrictive clause which is needed to precisely identify whatever "noun" it refers to).
You might still come across grammarians saying only that should be used for restrictive clauses, and only which for non-restrictive clauses, but hardly anyone thinks that nowadays - it's just pedantry.
You'll definitely come across plenty of definitions of what exactly which refers to in OP's construction saying it's a relative pronoun. Which they'll often say refers back to some earlier NP (noun/noun phrase).
But in this particular case the "thing" referenced by which is actually...
[The fact that] Oglethorpe served as chairman [is true]
That's to say which can refer back to a preceding statement, as well as a preceding NP.
Noting the comma in OP's example, consider...
I love chocolate, which makes me fat (I'm fat because I love chocolate)
I love chocolate which makes me fat (I specifically like the kind of chocolate that makes me fat)
In general, the presence of a comma (or a pause, in speech) before which/that indicates that what follows is a non-restrictive clause.
Some constructions are inherently ambiguous, but as a "rule of thumb", with non-restrictive clauses you should start looking for a credible statement or NP reference from the beginning of the sentence. With a restrictive clause, start looking for a credible noun reference immediately preceding the relative pronoun and work backwards until you find one.