This play was written in 1612, and uses archaic language. If you're going to read literature written in older English, you must consult dictionaries that include older meanings of words:
subtle (adj): 3. archaic Crafty; cunning
To properly interpret it in this context, it helps to include more of the scene:
ANTONIO. Now, sir, in your contemplation? You are studying to become a great wise fellow.
BOSOLA. O, sir, the opinion of wisdom is a foul tetter that runs all over a man’s body: if simplicity direct us to have no evil, it directs us to a happy being; for the subtlest folly proceeds from the subtlest wisdom: let me be simply honest.
ANT. I do understand your inside.
BOS. Do you so?
ANT. Because you would not seem to appear to th’ world; Puff’d up with your preferment, you continue; This out-of-fashion melancholy: leave it, leave it.
BOS. Give me leave to be honest in any phrase, in any compliment whatsoever. Shall I confess myself to you? I look no higher than I can reach: they are the gods that must ride on winged horses. A lawyer’s mule of a slow pace will both suit my disposition and business; for, mark me, when a man’s mind rides faster than his horse can gallop, they quickly both tire.
Bosola claims that he is not an usually wise or perceptive person, that he has modest ambitions and does not wish to think too deeply or to worry about having a quick wit -- in short that he merely wants to be "simply honest". In this context "the subtlest folly proceeds from the subtlest wisdom""the subtlest folly proceeds from the subtlest wisdom" is simply a warning that a man should not be too clever, as in the similar contemporary expression
You are too clever for your own good.You are too clever for your own good.
or
However in the full context of the play Bosola is putting on a false front, as he is actually the agent of the Duchess' brothers, sent to spy on her and her husband Antonio. He ingratiates himself with Antonio in order to gain his confidence, which later he uses to expose them.