The prepositions “by” and “to” represent very different relationships. Often, “by” indicates a means or an agency, but “to” indicates a target or a recipient.
. . . this decision was fully justified by economic conditions.
Here, “by economic conditions” represents the agent of the passive-voice clause. In the active-voice equivalent, agency is represented by the subject:
Economic conditions fully justified this decision.
Intelligent idlers justify their idleness to their intelligence.
This is an active-voice clause which is similar to the infinitive phrase in The French Lieutenant’s Woman. Here, the agent is represented by the subject. The phrase “to their intelligence” represents the target of the action or the recipient of the justification.
The intelligent idlers themselves and the intelligence of those idlers are two different things, playing two different roles in this clause.
The original context also includes a means, which could be introduced by the preposition “by”:
Intelligent idlers justify their idleness to their intelligence by setting their sights high.
Here, the intelligence and the act of setting are two different things, each with a different relationship to the verb “justify”. The intelligence is a target, and the setting is a means. Those are not interchangable relationships, and those are not interchangable prepositions.