If you want to be brief, the easiest thing to do is just answer the question.
Person A: What is the capital of France?
Person B: Paris.
Or if the question was buried in other statements and you need to make clear what you are answering:
Person B: The capital of France is Paris.
If the question is long and complex you don't need to restate the whole thing.
Person A: Considering what our production costs have been historically, do you think that we would be better off to import the parts that we need from Singapore or would it be a good idea to investigate the possibility of purchasing the machinery that would be required for us to produce these parts economically ourselves?
Person B: I think we should import the parts we need from Singapore.
If the nature of the back and forth is that you need to make clear that you are answering the person's question and not just supplying random facts or opinions, then by all means say, "The answer to your question is ..." or "To answer your question ...".
But in general I only use such phrasing if it is briefer than restating a long and complex question. Like:
To answer your question: No.
3 years after I wrote this post I got an upvote that brings my attention back to it. :-) And re-reading my answer, reminded me of a little anecdote.
When I was in school many years ago (yes, we had schools in the 1970s, though our textbooks were all written on clay tablets and gym class was mostly about learning to wrestle saber-tooth tigers), our teachers routinely said that answers to questions on a test must be complete sentences, and basically restate the question. So, for example, if there was a question that said, "When was the Declaration of Independence signed?", we couldn't just write "1776". We had to write, "The Declaration of Independence was signed in 1776."
My chemistry teacher apparently didn't know that all the other teachers were requiring this, and one day in class he went on a little rant saying, "When you answer a question on a test, you don't have to restate the question! I know what the question is -- I wrote it! Just give the answer. Like if I ask, 'What reaction takes place when you mix sodium chloride and magnesium?', you don't have to write, 'Yes indeed, there is a reaction when you mix sodium chloride and magnesium'. Just answer the question."
Personally I always thought the rule was silly. Like, what, was the teacher afraid she would forget what the question was? I suppose I could see it for an English class, where the point of the class includes learning to write grammatically correct sentences.
In normal conversation, we don't normally do this. If someone asks, "When will Fred be back in the office?", we typically say, "After lunch". Not, "Fred will be back in the office after lunch."