If you want to be brief, the easiest thing to do is just answer the question.
Person A: What is the capital of France?
Person B: Paris.
Or if the question was buried in other statements and you need to make clear what your answering:
Person B: The capital of France is Paris.
If the question is long and complex you don't need to restate the whole thing.
Person A: Considering what our production costs have been historically, do you think that we would be better off to import the parts that we need from Singapore or would it be a good idea to investigate the possibility of purchasing the machinery that would be required for us to produce these parts economically ourselves?
Person B: I think we should import the parts we need from Singapore.
If the nature of the back and forth is that you need to make clear that you are answering the person's question and not just supplying random facts or opinions, then by all means say, "The answer to your question is ..." or "To answer your question ...".
But in general I only use such phrasing if it is briefer than restating a long and complex question. Like:
To answer your question: No.