Refine means, in general, make finer. In this case what is made “finer”, in the sense “subtle” or “more penetrating” is Ishmael’s analysis of the Captain’s motives.
Let’ look at this in a little wider context.
THOUGH, consumed with the hot fire of his purpose, Ahab in all his thoughts and actions ever had in view the ultimate capture of Moby-Dick; though he seemed ready to sacrifice all mortal interests to that one passion; nevertheless it may have been that he was by nature and long habituation far too wedded to a fiery whaleman’s ways, altogether to abandon the collateral prosecution of the voyage. Or at least if this were otherwise, there were not wanting other motives much more influential with him. It would be refining too much, perhaps, even considering his monomania, to hint that his vindictiveness toward the White Whale might have possibly extended itself in some degree to all sperm whales, and that the more monsters he slew, by so much the more he multiplied the chances that each subsequently encountered whale would prove to be the hated one he hunted. But if such an hypothesis be indeed exceptionable, there were still additional considerations which, though not so strictly according with the wildness of his ruling passion, yet were by no means incapable of swaying him.
Ishmael is trying to find an explanation of why Ahab, although he seems willing to sacrifice all other purposes to the pursuit of Moby-Dick, nonetheless continues in the “collateral prosecution of the voyage”—commercial whaling. It could be mere professional habit, but there may be other, deeper motives. Ishmael has a couple of ideas: perhaps Ahab has extended his enmity toward Moby-Dick to all other sperm whales, or perhaps Ahab has made the mathematical calculation that the more whales he kills, the better the chance will be that the next whale he encounters will be Moby-Dick. But Ishmael prefaces these suggestions by admitting he may be “refining too much”—building his argument on a far-fetched, implausibly subtle premise.