Yes, both words share the notion of hurting those who hurt you. The differences are in the broader connotations and related ideas. "Retaliation" focuses on the action, while "vengeance" focuses on the result.
"Vengeance" is part of a broad notion of "justice." It plays into codes of justice common to many ancient systems of law (Norse, Babylonian, Mosaic) of "an eye for an eye," that if someone wrongs you, the wrong can be "paid for" by an equal wrong done to the other person. The notion of vengeance implies that this action reduces or resolves or offsets the wrong that was done to you.
Meanwhile "retaliation" just means that this action is in response to a wrong done to you. You could slap me and I could instantly slap you back. It can be independent of a "justice" system; my slap-back could be reflexive; I don't have to have felt "wronged." "Retaliation" just explains the motivation of the action, and it doesn't have to be because I felt that the wrong to me left an injustice that had to be paid for; it just has to be "in response." A retaliation of course could be part of "vengeance," but it could be an incomplete part. If you spread rumors about me at school, causing my boyfriend to dump me, then I might pursue vengeance first by spreading rumors about you in retaliation, but I won't consider my vengeance complete until I've first set you up with a boyfriend and then manipulated him into dumping you.