The official term for this under U.S. law is mental incompetence, and under UK law it seems to be mental incapacity, although it's likely both terms are commonly used.
Mental incompetence is the inability of a person to make or carry out important decisions regarding his or her affairs. An individual is defined as mentally incompetent if h/she is manifestly psychotic or otherwise of unsound mind, either consistently or sporadically, by reason of mental defect. A few examples of such a mental defect are retardation, schizophrenia, and dementia.
However (in the U.S.) you don't usually file a lawsuit for this issue. Instead (and note the common italicized legal terms), if you want to use this as a justification to gain control of their assets, you petition a judge to declare someone mentally incompetent. The law on this will vary from place to place, but a mental health professional will normally be involved in this process, who will perform a psychological evaluation (or mental health assessment) on the person, and then submit a report to the judge.
I don't think these kind of decisions ever involve trial, and certainly it's nothing a jury would decide. The case will involve one or more hearings during which the judge will review the evidence, and then rule one way or the other. If the court finds that the person is mentally incompetent, it can appoint a legal guardian to made decisions on their behalf. This may be a family member, or it may be a neutral third party.
In some cases if the person is deemed a harm to himself or to others, you may petition the judge to have them involuntarily committed, meaning that they will be placed in a psychiatric hospital or similar institution.
[Edit] Another answer has suggested power of attorney to describe this situation; however this is usually a process in which someone voluntarily grants someone else, often a legal professional, the power to act on their behalf. Power of attorney can be given for all kinds of reasons, not just mental illness, and is often limited.
Guardianship is often much broader in scope:
The guardian can be authorized to make legal, financial, and health care decisions for the ward. Depending on the terms of the guardianship and state practices, the guardian may or may not have to seek court approval for various decisions. In many states, a person appointed only to handle finances is called a "conservator."