My friend asked me, "Have you ever paid the money as a penalty?"
I paid some money as a penalty, but that was long ago; it won't effect the present.
What should I say?
I have paid.
I had paid.
Let's make up some details, so the difference between these sentences will be clearer.
Suppose that it is January, and last September your university sent you a bill charging you a $50 fine for registering late. They sent you another bill again in November, raising the fine to $75 because you were late paying it. If you still owe the money you will not be allowed to register today for this semester. Your friend asks:
Q1: Have you paid that fine? ... you answer:
A1: Yes, I have paid it.
You answer using the present perfect for two reasons: 1) Your friend's question used the present perfect, and we usually match the tenses in question and answer. 2) The present perfect expresses a past action which establishes a present state. You understand, because your friend uses the present perfect, that she is asking about your current status, and you answer with a statement about your present status.
Note, however, that if you want to tell her when you paid, you must employ the past tense, because now you are speaking of the event relative to that date:
Q2: Have you paid that fine? ... you answer:
A2: Yes, I paid in December.
Suppose now that your friend asks:
Q3: Did you pay that $75 fine? ... perhaps you answer:
A3: Yes, I paid it.
In this case your friend is not asking about your present status but about a past fact; so both of you use the past tense. However, it is also possible that you return a different answer:
Q4: Did you pay that $75 fine? ... and you answer:
A4: No; I didn't pay the extra $25 because I had paid the $50 fine in October.
Both of you are talking about your past action relative to the November bill, and you cast that in the past tense, as in Q3/A3; but when you add information about something which happened before November, you must cast that in the past perfect.
Since you are answering a question, which uses the present perfect, your answer should be "Yes, I have." or "No, I haven't." You could also simply answer with "Yes." or "No."
If then you want to add when it happened, you can add "It happened time ago."
The question is incorrect. It should be:
"Have you ever paid money as a penalty?"
In fact, it should be:
"Have you ever paid a fine?"
(fine (noun): "3 a : a sum imposed as punishment for an offense b : a forfeiture or penalty paid to an injured party in a civil action"
Your answer should be:
Yes, I have.
Yes, once I paid a fine as a penalty, but that was a long time ago. It shouldn't be important now.