No, they are not the same.
You asked me if they came or not.
unequivocally asks about whether a particular event in the past happened or not: either they came, or they did not. (Of course I might not know the answer).
You asked me if they were coming or not.
also asks about something in the past, but it could be asking two different things: at the particular point in the past which you are discussing (assumed to be known to both of you) were they on their way here ("were coming" in the continuous sense); or alternatively, were they intending to come ("were coming" in the "future intention" sense). The actual coming, if it happens, might be in the past, or still in the future.
You asked me if they would come or not.
This is different again (though in some contexts it has the same meaning as the second option). "Would" is 'future in the past' - You asked me "will they come or not?" - asking for a prediction, or possibly a question about their intentions.