The point of economic rationality refers to the level at which spending money for any particular purpose will bring a sensible return. In other words, it refers to the point at which one gets value for money.
It is not economically rational to spend a large amount of money on a project when exactly the same results can be obtained far more cheaply. People who lavish unnecessary money on a project generally have a motive other than getting value for money (whether in bribes, kickbacks, favours or family considerations).
What your statement is saying is that banks that take vigorous legal action against fraudsters discourage other potential fraudsters from attacking them. In order to achieve this goal, such banks are prepared to spend a great deal of money in the courts, far more money than be justified economically.
It would cost these banks less to suffer the frauds than to pay the excessive legal costs. But they are prepared to pay large amounts to frighten off other fraudsters.