Pulling out fingernails is an ancient form of torture. It's not all that common an idiom, but it would refer to doing something very unpleasant to someone to force them to do something. Normally it is used as a hyperbole for situations where you really had to pressure someone
I had to pull his fingernails out to get him to tell me why he didn't want to go to her birthday party.
"Pulling teeth" is an expression that is used in a simile to suggest that something was difficult or required an unexpectedly large amount of effort
Getting him to tell me why he didn't want to go to her birthday party was like pulling teeth.
Note that in the latter case, the expression describes how difficult it was for you to extract the information. In the former, it describes how unpleasant the situation was for the other person.