This sentence is a little tricky for two reasons. First, the word "help" doesn't quite mean what it normally means and, additionally, the words "Dark Arts" interrupt the phrase "nearer than".
Let's first define "help" in this context. There is a common enough phrase "can't help" which means that something is beyond the person's control. You would usually find this in a situation where someone accuses someone of doing something wrong, and the accused responds with "I can't help it". Here is the Cambridge Dictionary's definition:
to not be able to control or stop something:
"Could help" is essentially the opposite. It expresses the idea that something is in fact within the person's control. Now we can see that the sentence is saying that Dumbledore was trying to keep Snape away from the Dark Arts. He allowed Snape to come no nearer than he could help, i.e. he kept him away to the fullest extent that he had control over.