Our English teacher has explained to us that when "that" is used in a relative clause, which is not a non-defining relative clause, which means I can't use a comma, but when I studied some examples, I found the following :
It's hoped that the peaceful use of every bit of science would bring in an age of prosperity, that has never been seen before.
This looks like an error.
The source is a student essay. The original has:
It is hoped that the peaceful use of this tremendous energy would bring in an age of peace and prosperity, such as the world has never been before.
That's quite well written (although the use of "would" instead of "will" is questionable).
This has been mangled by someone who wanted to adapt it for a comprehension test, and the resulting sentence is ungrammatical and rather meaningless.
That sentence is indeed, at least from a "prescriptivist" point of view, incorrect. These rules aren't generally known, at least consciously, by native speakers, however, and they are often broken, and there are some who say they aren't "real" rules. So while it's still a good idea for you to learn them so you know the formal rules and because following these rules will reduce ambiguity, but you shouldn't expect them to be followed in general.
There are two main ways this sentence can be read:
It could be read a restrictive clause ("not a non-defining relative clause" is just a more complicated way of saying "restrictive clause"), in which case it should not have a comma: "It's hoped that the peaceful use of every bit of science would bring in an age of prosperity that has never been seen before." This phrasing would mean "It's hoped that the peaceful use of every bit of science would bring in an age of prosperity and that that age of prosperity would be one that has never been seen before." In this reading, "that" is a coordinating conjunction.
The other reading would be that it's an independent clause. Independent clauses should be set off with a semi-colon or a period/full stop: "It's hoped that the peaceful use of every bit of science would bring in an age of prosperity; that has never been seen before." In this reading, "that" is a pronoun that appears to refer to "an age of prosperity", but could refer to "the peaceful use of every bit of science bringing in an age of prosperity". Depending on which it's read as, the passage would mean "It's hoped that the peaceful use of every bit of science would bring in an age of prosperity. An age of prosperity has never been seen before." or "It's hoped that the peaceful use of every bit of science would bring in an age of prosperity. The peaceful use of every bit of science bringing in an age of prosperity has never been seen before."
There is a third reading that requires a bit more of a stretch than the other two. That reading is that it's a non-restrictive clause. In that case, the comma is appropriate, but the clause should be introduced by "which" rather than "that": "It's hoped that the peaceful use of every bit of science would bring in an age of prosperity, which has never been seen before." This would have a meaning very similar to the second reading.
BTW, you actually made a similar error in your question itself. In "Our English teacher has explained to us that when "that" is used in a relative clause, which is not a non-defining relative clause", the phrase "which is not a non-defining relative clause" is a restrictive clause , and so it should be introduced by "that" without a comma: "Our English teacher has explained to us that when "that" is used in a relative clause that is not a non-defining relative clause...". By putting a comma after "relative clause", you make what comes after a non-restrictive clause, which means that you are asserting that "is not a non-defining relative clause" is a property that all relative clauses have. That is, it makes "is not a non-defining relative clause" a parenthetical remark describing relative clauses in general, rather than specifying that you are talking about a particular subset. Your sentence would be even clearer if you just used this phrase an adjective modifying "relative clause": "Our English teacher has explained to us that when "that" is used in a non-restrictive relative clause..."
