Can we use the comma before "the other"? I think it should be replaced by a semicolon or a period.
Newton has two cats. One is big, the other is quite small.
Yes, this is absolutely correct and you may write the sentence that way. Instead of a comma you may also use "and". That is:
Newton has two cats. One is big and the other is quite small.
Either is perfectly acceptable. The comma simply replaces the understood "and". No semicolon or period is "missing" there (though you could use a period there, it would just break it up into very short sentences. I prefer the original version you gave.) You could also link the two sentences with a semicolon if you like:
Newton has two cats; one is big, the other is quite small.
Any of these variations and your original sentence are all acceptable. You may use any of them.
I would rather use the punctuation as follows.
Newton has two cats: One is big and the other is quite small.
I used the colon because what follows is an a description of two cats. Since what follows the colon is a sentence, I capitalized the first word. I didn't add any comma before the other, since I am using and.
I would have used the semicolon in a sentence like the following one.
There are two things I like: walking on the beach an hour before sunrise, when the breeze caresses my face, and the beach is still quiet; drinking beer with my friends while we watch the sun setting, the people closing the windows of their houses, and my cat fighting with my neighbor's cat.
Since what follows the colon is a phrase, I didn't capitalize the first word after the colon, which what I would do in the following sentences too.
There are three insects I don't like: mosquitoes, fruit flies, and ticks.
I like three things: eating on the morning, walking on the beach, and chatting with friends.