"Flattering the professor" is saying things to the professor that make him or her look good. Like saying how excellent his latest publication is, how nice his or her haircut is, and so on. "Flattering" can be truthful and deserved - if the professors latest publication was excellent, there is nothing wrong with saying it. Or it can be not quite truthful. Or it can be completely untruthful, which often backfires. However, there is a connotation that the flattering is done to get into the professor's good books, which would be a slight character flaw in yourself. But only a slight one.
"Sucking up to the professor" means you are saying things solely to get into the professor's good books, and whoever said this thinks that this is detestable and a huge character flaw on your side. There is "sucking up" that is not at all flattering - if the professor does something wrong and you loudly agree, that's not flattering. Or if the last publication was condemned by critics, and you say "I don't care what they say, to me it was excellent", that might be "sucking up" but not flattering.
So "flattering" and "sucking up" are somehow related, but there are huge differences.