Either form could be correct, depending on the context. If you are writing this sentence and trying to choose the correct verb agreement, then your choice depends on your intended meaning. If you are reading the sentence, then assume that the verb is correct and derive the meaning from its agreement. To illustrate the difference, I will interpret both sentences assuming that they are correct:
Painting involves a fixed perspective and a form of rational thinking that engages sight.
Here engages is singular, so it must refer to a form of rational thinking.
Painting involves a fixed perspective and a form of rational thinking that engage sight.
Here engage is plural, so it must refer to a fixed perspective and a form of rational thinking together.
As was pointed out in the comments, the last form you wrote is incorrect, because "that" should be used with essential relative clauses, which should not be offset with a comma (unlike this nonessential relative clause using "which").