My question is should I be using 'would' to reply as the question is asked with 'would'?
No, the use of the word would in the question doesn't force the use of would in the answer. Your first answer is fine.
Furthermore, the original question can be interpreted in one of two ways: either the person is really asking for your bike, or else the person is merely posing a hypothetical scenario.
In the latter case, if you wanted to keep the conversation in the realm of the hypothetical, then you'd use wouldn't instead of won't.
It's also worth noting that the "mismatch" can go the other way around, too, and the conversation still sounds normal:
Bob: Will you give me your bike?
Rob: No, I wouldn't give you my bike.
In that case, Rob is more or less saying that the answer is no, and that he can't think of any possible reason that his mind could be changed.