Formally there is more than one interpretation. Pragmatically only a couple of readings are possible.
While it could mean "driving by the accident", that doesn't seem likely. Fire engines and ambulances drive to the accident, they don't usually drive past it. That interpretation could be possible if the point you are making is "The ambulance didn't stop!"
There was an accident and some people were hurt. We were all relieved when we saw the blue lights of the ambulance, but it just drove by. Perhaps there was another crash or some other emergency. Anyway, we had to wait another 20 minutes for our ambulance.
It is far more likely that it means "driving by me". What it doesn't say is whether the ambulance was going in the same direction or the opposite direction. It is even possible that the person is just sitting by the side of the road watching and not in car. Other context might make it clear.
We were stuck in a queue on the freeway when we saw an ambulance driving by. It took another hour and thirty minutes before the police reopened the road. It must have been pretty bad. (implicitly the accident is ahead, and the ambulance is going in the same direction, towards the accident.)
It is also possible that the ambulance could be on the other carriageway going in the opposite direction.