As far as I understand, in American English there must stand that instead of which in the sentence
"Of these two birds the male is that which is colored brighter"
the clause being restrictive. On the other hand, "that that" is definitely not an option. Does it mean that the sentence is impossible in formal writing and must be reworded?
There's nothing wrong with that which here.
You are mistaken in your belief that that must be employed with restrictive relative clauses: both that and wh- relatives may be used in this context.
The idea of employing only that with restrictive relatives was first advanced in 1851, at a time when grammar-writers were inclined to rationalize the language. It was given wide currency by the Fowler brothers' The King's English, which argued that "[I]f we are to be at the expense of maintaining two different relatives, we may as well give each of them definite work to do", and by the elder Fowler's even more influential Modern English Usage. It was subsequently adopted by some fairly reputable style guides.
But it is not a rule in any register, formal or informal. Some people follow it, others do not; and even those who follow it acknowledge many situations where it not only may be suspended but must be. Fowler himself acknowledged that "[I]t would be idle to pretend that it is the practice either of most or of the best writers."
"That that" is acceptable grammatically, but it doesn't "sound good", and so should be avoided. Native speakers would probably prefer to avoid even coming near that construction, saying instead
"of the two, the male is more brightly-colored"
or
"the male is the one with the brighter coloring".
There is a lot of formal phrasing in English that people avoid using because there's no benefit to it -- it sounds stilted, bookish, and, [edit]sometimes, especially when used in everyday conversation[/edit], as though the speakers are "giving themselves airs" [pretending to a higher social status].
MMacD is correct that some of the language sounds "formal" but I think mistaken to suggest people who talk this way are "putting on airs".
A native speaker might use these structures in academic writing, which is naturally fairly formal. Moreover, academics might write this way because it mirrors their train of thought -- in other words, they are writing as they think something through, and then force the grammar to fit their thought, rather than rewriting the entire sentence.
This is more common when speaking than writing, since you can't revise a speech or a lecture. But it also indicates what is important to the speaker, and how they derive or deduce some conclusion, with the way they structure their argument.
For example, suppose I'm talking about butterflies and moths in an academic setting:
When speaking of the Order Lepidoptera, the taxonomy is not as structured as many would like, since of the two groups that comprise the order, the butterfly is that which is monophyletic, while the moths have a more diverse phylogeny.
Of course there are shorter and more direct ways to phrase, this. But also keep in mind that some academic speakers, although undeniably intelligent, might not care if their sentences are clear and concise. Their audience should still be able to follow along and understand their meaning.
