"That" can be a pronoun, determiner, adverb, or conjunction. When it's used as a conjunction, it does need a complete clause. In its other meanings, it doesn't. For instance "Is that hot?" is perfectly fine.
Now, when it comes to picking between A nd C, A I thought could be eliminated because it doesn't encloses the non-essential information in a comma pair.
I don't think "essential" is the best term. The question is whether it is restrictive. The phrase "living along the West Coast of North America" is restrictive; it restricts the subject from sea otters in general to a subset of sea otters. Restrictive clauses should not be set off with commas.
Does it means that all the sentences which come after that need to be a complete clause with a subject and a verb?
When you're talking about a word, rather than using the word, it should be set off from the other words somehow, e.g.
Does it means that all the sentences which come after "that" need to be a complete clause with a subject and a verb?
I don't know of a way to do sentence diagrams in SE, so I'll just use parentheses:
It has long been known (that the sea otters (that live along the West Coast of North America, (where they help keep kelp forests in their habitat healthy and vital)))
Here, "where they help keep kelp forests in their habitat healthy and vital" is a subordinate clause to "the West Coast of North America".
"(that live along the West Coast of North America, (where they help keep kelp forests in their habitat healthy and vital)))" is a restrictive clause modifying "sea otters". So the entire passage "the sea otters (that live along the West Coast of North America, (where they help keep kelp forests in their habitat healthy and vital)))" is a noun phrase; it introduces the phrase "sea otters", then further describes what sea otters they're talking about. The important part is that In Option $D$, everything that follows "sea otters" is modifying "sea otters". While there are verbs in that description, they are in subclauses, and none of them have "sea otters" as a subject. So for the purpose the analysis, we can replace "the sea otters ..." with "certain sea otters".
Now the sentence becomes "It has long been known that certain sea otters". That's the entire sentence. Remember, everything else in the sentence is just saying which sea otters they're talking about. This then is a sentence fragment; it doesn't present a complete thought that can be the object of the verb "known".