This is a simple interjection. Thoreau is expressing exasperation.
This is an elision. The full expression is "What I would not give for...", meaning "I really want..." Thoreau is shortening it to just the last word.
a man who is a man,
that is, a man who is a real man, and not a "unprincipled foreigner or hireling native" (kinda racist, but what are you going to do?)
and, as my neighbor says,
His neighbor uses a colloquial expression, which he will now quote
has a bone in his back which you cannot pass your hand through!
In modern English, we say a man has a "spine" or "backbone"; figuratively, he is stalwart and courageous. The imagery here is that a weak man's backbone is not missing, but is insubstantial, like a column of smoke or a stream of water, and you could put ("pass") your hand through it.