Sounds like a case where you could use whom instead of who:
you were the one whom I talked to yesterday, weren't you?
Whom should be used to refer to the object of a verb or preposition. When in doubt, try this simple trick: If you can replace the word with “he”’ or “’she,” use who. If you can replace it with “him” or “her,” use whom.
So as you're not asking who was talking but to whom you were talking "whom" would probably sound better.
Also some minor corrections (off-topic):
and had talking about something
"had talking" is a combination that to my knowledge doesn't exist. It's either "and we talked about something" (simple past), or "and we had talked about something" (past perfect) or "and we have been talking about something" (past progressive) or ".. had been talking.." (past perfect progressive). I'd probably stick with the simple past as it's not related to other events in the past or present (similar to the "met" in the first part of that sentence).
i wasn't sure that he was the same person.
Also "if" or "whether" might be better suited than "that" in that sentence.