Consider that neither of your given responses directly answers the question of where you have been. A direct answer would be, "In the garden." Instead, you have (correctly) assumed that the question implies more knowledge than simply your prior location. For example, why were you there?
If you understand what kind of information the question is intended to gather, then almost any answer that gives that information will be seen as correct.
I just finished cleaning the garden.
I was at the bank.
I was playing with my son at the park.
I did my homework then took out the trash.
I had to go to the dentist.
They are all perfectly acceptable responses (depending on the context), even though the tenses might not match up. And if that doesn't tell the person what they need to know, they will tell you.