a. I talked to the man dancing. [He is the dancer.]
b. I talked to him dancing. [He is the dancer.]
In written, formal English, the man is dancing, not I. If you wanted to say that I was dancing while talking to the man, you would need to add a comma before dancing, like this:
a. I talked to the man, dancing. [I am the dancer.]
b. I talked to him, dancing. [I am the dancer.]
Please keep in mind, a lot of native speakers don't necessarily know or honor this rule and rely solely on the context of the sentence to understand its meaning. Also, while this example above is now saying what you intend it to, it still might seem awkward to people simply because they wouldn't communicate in such a potentially misleading way. The best way to keep sentences clear is to move modifiers as close as possible to the nouns they modify. For example:
a. Dancing, I talked to the man. OR: While dancing, I talked to the man.
And now the final sentence:
c. I talked to him while dancing. [I am the dancer.]
In this sentence, while is a conjunction that connects two events that are happening at the same time. (You can read about while here: http://speakspeak.com/resources/english-grammar-rules/conjunctions/while)
While dancing is an example of a reduced adverbial clause. It's a shortened way of saying while I was dancing. It's I and not him because you can't say him was dancing. Pretty subtle. (You can read about reduced adverbial clauses in this stackexchange post: Use of -ing after while)