I think order there means the instruction of the receiver of the check. The recipient of the check must endorse or sign it for it to be cashed. The phrase pay to the order of is contrasted with pay to bearer, which doesn't require an endorsement by an individual, as explained here:
Investopedia pay to order
What Is Pay to Order?
Pay to order describes a check or draft that must be paid via endorsement and delivery. Pay-to-order instruments are negotiable checks or drafts that are generally written as "pay to X or pay to the order of X." The name entered here indicates the specific person, group, or organization that the payer authorizes to receive the money. Pay-to-order instruments stand in contrast to pay-to-bearer instruments, which do not require an endorsement.
[emphasis added]
However, you can't just plug in a synonym of order and expect the whole phrase to mean the same thing. It's a fixed phrase with an accepted legal meaning.