The reason here is because the it in the answer is referring to the subject being queried in the question - which is this, which has indeterminate (neutral) gender and is hence an it - not your mother (which has female gender and hence would be she)
Is this your mother?
No it isn't.
Is Hannah your mother?
No she isn't.
Is David your mother?
No he isn't.
If we change the subject of the question to be your mother we can see that this changes the response to use she. And note also that if we expand the response to include a reference both to the subject and to the object of the question, we can see that the pronoun referring to the object takes the gender of the object, and the pronoun referring to the subject of the sentence takes the gender of the subject.
Is your mother bringing the car?
No she isn't.
No she isn't bringing it.
Is your mother bringing Hannah along?
No she isn't.
No she isn't bringing her.
Is your mother bringing her new boyfriend along?
No she isn't.
No she isn't bringing him.