Neal had both hands on the counter, clenching the muscles in his forearms.
Fortunately, thanks to the use of, once again, context, we can actually infer what the word means. You will have to do this frequently when reading English because English authors will often use words in unconventional ways that may not even be listed in a dictionary. Asking a question on Stack Exchange each time you come across something like this in English literature is impractical.
Neal had both hands on the counter.
We know from this that Neal has both of his hands on the counter, and isn't squeezing them together. It also mentions that "his head was hanging down." The mental image portrayed here is of a very common gesture (at least in Western culture), with arms out and apart, supporting one's body as they lean over a counter, looking down. The gesture is associated with tension and distress; tension in the mind often directly relates to physical tension in the body.
While I admit that "clench" is perhaps not the ideal word for this sentence, "clenching" is characterized by a tightening of the muscles, because it is required in order to squeeze your hands, teeth, etc. together firmly. Therefore, based on context clues, it makes sense to infer that clenching means the same thing as 'flexing' or tightening the muscles in this context. It is likely that Neal is pushing against the counter in stress or frustration.