Another way of saying that a person is on a road is "the road below him". We could move the phrase to be immediately after the road and it would have the same meaning, and would be a little clearer:
The road below him dropped twenty feet down to a little stream.
The phrase "down to a little stream" means "downward towards a little stream".
To drop can mean "to descend". When using this phrase of roads (and not of freight elevators, say) the descent is on a slope.
The road on which he was traveling descended twenty feet to a little stream. Presumably he crosses that stream.
I don't think we can be as confident as Peter is that the distance referred to here is vertical. The stream could easily be twenty feet ahead (down the slope, down the hypotenuse, as it were) and the same words could be used.
Now, if the road was making a kind of steep corkscrew turn (it might be, we don't have enough context to say) then Peter could be correct in his understanding that the vertical distance is being referred to here.