With hypothetical statements, logic is as important as tense. This hypothetical scenario is about a past action, as it asks "what would you have done" (as opposed to 'what would you do' in the future). You might think that it sounds correct to match the tense by qualifying it with "if you had been in my place", as we use "were" for present scenarios (eg "If you were in my place, what would you do?").
It depends on how you want to describe the scenario. Some hypotheticals are 'impossible' - for example, "if I were you...". It isn't possible to be someone else, but that is part of the hypothetical. However, to ask what someone would do "in your place" is not necessarily impossible - something you undertook could well have been done by someone else instead of you, so asking "what would you have done if you had been in my place?" is correct because, in this hypothetical scenario, both events are in the past (the thing that was done, and the other person taking your place). However, if you are using the classic hypothetical scenario where you ask what the other person would have done, or do, if they actually were you, the tense doesn't really matter so much - if a person were you, they would still be you.
There are other, perhaps simpler, ways of saying it:
- What would you have done in my place?
- What would you have done instead?