So, let's look at the sentence, so we can start pulling it apart:
You almost couldn't design a worse problem as a fit with our underlying psychology or the way our institutions make decisions.
The principal verb here is could, which is negated as couldn't. The subject is you, and the object is the verb complement, "design a worse problem...".
So, this parses as
You couldn't (design a worse problem as a fit with our underlying psychology or the way our institutions make decisions)
I've excluded the almost because it is simply there to weaken the couldn't. If "you couldn't" do something, it is impossible to do that thing. If "you almost couldn't" do something, it is almost impossible to do that thing. So, this means that it is almost impossible to:
design a worse problem as a fit with our underlying psychology or the way our institutions make decisions.
The as introduces a prepositional phrase. In this case, it could refer to the properties one might be attempting to design into the problem. Taking it in context, it applies refers to the worse - the ways in which the problem would be designed badly.
(design a worse problem) (as a fit with (our underlying psychology) or (the way our institutions make decisions))
As a fit with refers to how well the problem 'fits' into things - or how well suited those things are to the problem.
Thus, in this hypothetical idea of designing a problem (which isn't something we do in real life), we would be trying to make the problem not fit "our underlying psychology" and make it not fit "the way our institutions make decisions".
our underlying psychology
That noun phrase simply refers to the way our (people's) minds work.
the way our institutions make decisions
That noun phrase refers to the processes or methods (the way) used by major organisations (instutitions) to decide things (*make decisions).
Taken as a whole, then, it says that it would be almost impossible to invent a problem that would be a worse fit for the way our minds work and for the way major organisations make decisions. This is an idiomatic way to say that the problem is not something that we are suited to solving, given the way our minds work and the way organisations make decisions.