from The BBC Learn English: We don't use to + the infinitive with 'not'. For example, I wouldn't say "I went to bed not to watch TV." That doesn't make sense. I would prefer to use a different verb to express the negativity. For example ''I went to bed to avoid watching TV''. However, we do use 'not' and the infinitive when we make a contrast with 'but', so for example, "I went to bed not to sleep, but to watch TV."
IELTS Liz uses to + infinitive in an examplary essay to provide a band 9, which is the maximum score, example. I can't provide you with the video as it's a paid one so there's no public link. But she wrote this essay as a 'perfect one' not a 'rarely perfect one'.
So is it correct to write to + infinitive with not?