People understand Ask away by analogy with certain other familiar sentences with away.
Soldiers shout this when dropping bombs from an airplane (see, for example, this book):
Bombs away!
By itself, the word away means "located somewhere else" or "at a distance". In the phrase bombs away!, it suggests movement: "going somewhere else". Bombs away! announces that the bombs have just been released and are now in motion, though many non-soldiers think it's a command to release the bombs. Either way, it refers to the bombs' being "on their way" to their destination, freed from the plane, moving away from the plane, soon to cause damage when they hit their target. A similar sentence is Torpedo away!, said after launching a torpedo (example).
Soldiers say this to mean, "Start shooting your guns, and don't hold back":
Fire away!
Fire here means to shoot a gun. Here, away has the same connotation as in Bombs away!: that you are releasing something that will fly "away" and do damage. It also suggests immediacy and a lack of inhibition. You should eagerly "release" your potential firepower "right away" and hold nothing back.
A metaphorical usage of Fire away! appears in this well-known song from 1980: "Hit me with your best shot—fire away!" The literal meaning is: "Hit me as hard as you can. I am tough enough to withstand it." The speaker is confidently challenging the listener to a fight.
People commonly say Fire away! metaphorically to mean "Ask me all the questions that you want to ask", especially when these questions might be difficult for the speaker to answer, either because they might dig into something embarrassing to the speaker or because the speaker might not know the answer.
When you say:
Ask away!
you are telling your listener to ask any question or questions that the listener wants to ask. The nuance of meaning comes from the way this sentence echoes sentences like those above (and some others, but hopefully that's enough). It suggests that the listener should not hold back or be inhibited about asking these questions, even though the questions might be difficult or upsetting for the speaker.
Ask away! doesn't necessarily suggest that the questions will hit the speaker like bombs or bullets. But this sentence is especially appropriate in situations like a meeting where the speaker must answer hostile questions about something unpleasant or embarrassing, like the "rapid-fire questions" mentioned in this difficult situation. In any sort of context, Ask away! suggests that the speaker is confident that he will be able to address whatever the listener "throws at him". It can also be a way to reassure the listener that the speaker won't take offense to a question.
Notice in the story that the listener, Slughorn, is upset by the question. He suddenly switches from affable confidence to fear. The question appears to hit him pretty hard.