You don't need to capitalize the o in the word "one" if it is not the first word of a sentence. The tutorial you watched is correct. The reason the first sentence is ungrammatical is because it is a run-on sentence, which is a very common grammar error. A run-on sentence has two or more parts (clauses), each of which can be a stand-alone sentence. (See how my sentence is similar to the sentences are you asking about?) To avoid making run-on sentences, you need to connect these clauses with conjunctions, such as "and", "or", "but".
So the problem with the first sentence is that "them" is not a relative pronoun and the clause is an independent clause. You need a conjunction to connect the two clauses if you want them both in one sentence.
I have two assignments, and one of them is done.
Or alternatively you need to make them two separate sentences, which means you need to replace the comma with a period.
I have two assignments. One of them is done.
The second sentence reads fine as long as you follow the correct sentence case and change "One" to "one".
I have two assignments, one of which is done.