You'd have to put it relative to some other landmarks on the actual street where he needs to be waiting.
E. g. if this is in the middle of a block, and there is a coffeeshop on one corner of this block (to the right of X) and a bank called Super-Duper Bank on the other corner of the block (to the left of X1), tell your friend to stand
just a few meters closer to the Super-Duper bank.
There are phrases like "up the street" and "down the street", and they might refer either to the North-South direction or to which end of the street is considered its "beginning", but in reality, any "up" or "down" will depend on which side you are looking at it from. So you'll need to explain it in terms of how it is relative to some specific known locations. "Between the pharmacy and the Gap store", or something like that.
P. S. If you actually know that your friend will be approaching points X1 and X from the left, you can say "don't go as far as.... [e. g. a pharmacy that is right where X is]". When I give directions, I usually say, "If you have reached X (a pharmacy), you have already gone too far. Go back a few meters."