A guy had a tree mushroom that grew in his lungs and then it caused a brain abscess. (split gill mushrooms, consumed in many countries). He was on corticosteroids for something, so i wanted to say "it's uncommon, he wasn't the strongest tool in the box" that's probably wrong though, what fits better than "the strongest tool in the box"
Maybe "he wasn't exactly a prime specimen".
This is vernacular (slang) and may not be understood in all regions. Of course if you want to be as clear as possible, avoid slang and just say "he wasn't exactly in prime physical condition" or "... in perfect health."
You may be thinking of "not the sharpest tool in the shed" (or variants of that) but this refers to intelligence rather than physical health.