Unquestionably, OP's first example is "standard/correct" for what I assume is the intended meaning (finishing this work is something that's impossible for me to do myself).
But that's not inherent in the context of constructions involving impossible [preposition] me and an activity expressed using an infinitive form. Consider, for example,...
1: It seems impossible for me to put a man on Mars.
2: It seems impossible to me to put a man on Mars.
Both versions are grammatically valid, but they mean different things.
In #1, for me modifies impossible. Perhaps someone else can put a man on Mars, but apparently I can't.
In #2, to me modifies seems. Perhaps putting a man on Mars seems possible to [some/all] other people.
Note that (probably for semantic reasons), when for me is used in close proximity to a verb like seems, there's a strong tendency for us to interpret them in conjunction. Consider the situation if we slightly reshuffle the word order in #1 above...
1a: For me it seems impossible to put a man on Mars.
1b: It seems for me impossible to put a man on Mars.
I'll admit straight away that #1b is a bit clumsy, but it's not really "ungrammatical". The point is most people would interpret both as semantically equivalent to #2, when for me is moved closer to it seems.
Having said all that, I do not believe all native speakers would always make this semantic distinction in all related contexts. That's to say there's a certain amount of ambiguity that cannot be fully resolved using "rules of grammar".
It often happens in such situations (i.e. - where different prepositions are at least "logically" credible) that if there are also different possible interpretations, idiomatic preference gradually settles on consistently assigning each different interpretation to one particular preposition.