There is nothing grammatically wrong with
'He is a man whom I look up to.'
The relative clause is non-defining, so no comma;
the relative pronoun refers to a person, so 'who', not 'which';
the relative pronoun is not the subject of the verb in the relative clause but its object, so 'whom'.
But, apart from the fact that 'whom' is rather formal, rather old-fashioned, it is not quite right either? Why?