All of the following assumes that these are in the past tense rather than the subjunctive mode. In other words, we're talking about actions that failed, rather than speculation about possibilities.
Both James and Sean couldn't do it.
From this sentence, we have no idea whether James and Sean acted together or separately.
Neither James nor Shawn could do it.
From this sentence, we know that they acted separately.
James couldn't do it, nor could Sean.
They acted separately, formally, and possibly half a century ago.
James couldn't do it. Neither could Sean.
They acted separately and we've stopped sounding old-fashioned.
Not James and not Shawn could do it.
It looks grammatically possible and I understand some of the logic behind it, but...it's just...weird. I'm not sure how to interpret it and I can't imagine the circumstances that'd make it sound natural.
Were the "could" subjunctive, I might think that anyone but those two might do it.
Not even James or Sean could do it.
They acted separately, and we seem surprised that they failed.
Not even James and Sean could do it.
They acted together, and we still seem surprised.