The cited example is nothing to do with "past tense" as such - it's a subjunctive usage.
English basically only has two verb tenses (Present and "Not Present"). In the general construction If you were X then [something would be true], were is an example the subjunctive mood. It doesn't refer to what you were in the past, but to some hypothetical / unreal / "not here-and-now" situation (often called an irrealis reference).
OP's brother is completely mistaken in thinking could is any more appropriate than would for actual past tense contexts. Twenty years ago I could / would cycle to work every day is perfectly valid with either verb form. They just mean different things (could = I was able to do that, would = I usually did that).
The could/would distinction is also irrelevant to subjunctive contexts such as OP's. Consider these simplified examples...
1: If I were strong I would beat you.
2: If I were strong I could beat you.
...where since the subjunctive refers to an "unreal, untrue" situation, it's pragmatically implied that I'm not strong. And #1 asserts that in some hypothetical situation where I was strong, I would definitely beat you. But #2 simply asserts that in that scenario I would be capable of beating you (but we might never fight anyway).
OP's brother is basically saying Because you are kind, I will let you pass by me. Which he contrasts with the hypothetical possibility of you not being kind (if that were true, he would not let you pass). It's syntactically perfectly valid for him to say If you were unkind then I could not let you pass, but this would mean something different (as with my earlier example, could would imply he had the ability to let you pass, even though he might not in fact do so.
Things can get even more complex with modal forms such as could in such contexts. The likely implication of If I had the money I could lend it to you is that I really would lend you the money in that hypothetical situation, so effectively could and would are interchangeable there (and both are in fact used). But if the word could were to be heavily stressed in that example, this would unambiguously convey that I'm only talking about being able to lend, not actually lending...
3: If I had the money then I could lend it to you. But I wouldn't anyway, because I don't believe you would ever pay me back.