This is what I wrote
Where c1, and c2 are the terms the system is trying to find the similarity between.
is using 'between' at the end correct?
I would be inclined to go the more direct route:
... where c1 and c2 are the terms whose similarity the system is trying to find.
This structure keeps the emphasis on the terms and eliminates the dangling preposition.
Per @ColleenV, please note that while "whose" is typically used to refer to an animate object, English does not have an inanimate pronoun to handle this situation, so "whose" is correct.
... where c1, and c2 are the terms the system is trying to find the similarity between.
I am guessing that this clause is used following an equation or a clause containing c1 and c2. [If this is so, please edit your question to make this clear]. I have indicated that the equation or clause is missing in the usual way, using three dots: this is called an ellipsis symbol.
In your clause, between is a preposition. If you end a sentence with a preposition, it's called a dangling preposition. Some people don't like dangling prepositions and write very complicated sentences to avoid them. But if the sentence is more understandable that way, let that preposition dangle.
In this case, there is a very simple way of avoiding the dangling preposition. Just re-arrange the clause like this:
... where the system is trying to find the similarity between [the terms] c1 and c2.
Just add a word for making it an understandable (math) sentence.
Where c1, and c2 are the terms, the system is trying to find the similarity between them (both).