No, you cannot switch 'that' to 'than' in this example. We use 'than' in comparisons with a comparative adjective, such as richer:
Today, humanity is in a richer state than it has ever been in.
Your example uses the superlative, 'richest', so there is no comparison required - the superlative is greater than all others.
We use 'that' to introduce a defining clause - one that adds further, essential information - although it can be omitted and the sentence is still grammatically correct.
- Today, humanity is in the richest state that it has ever been in.
- Today, humanity is in the richest state it has ever been in.
define:that
summarizes a bunch of info from dictionaries, and includes the relevant definition in under the fourth form (conjunction): "introducing a subordinate clause expressing a statement or hypothesis". I use that search operator all the time.