Google is a great tool, and it's smart too. Have you ever noticed that whenever you make a spelling mistake it tries to correct you, and suggest valid alternatives?
Just by typing passer, Google provided a few suggestions. When I typed the OP's expression in bold passer byes Google opened the page with
So to answer the OP's question, first check the spelling. The singular form is written passerby, but (and this is the interesting part) the suffix -s must be attached to passer and not to the preposition by. Therefore, the plural form is passersby, NOT passerbys.
I would re-write the OP's sentence as
The majority of people in our lives are just passersby.
Most of the people in life / in your life are just passersby.
The majority of people in your life are just passing by.
The phrasal verb, to pass by means to go past
“She would beg from the people passing by”