Imperative: She said to him "Buy milk"
Infinitive: She told him to buy milk.
Couldn't it have been, She said to him to buy milk
Although said to him doesn't sound ungrammatical, it doesn't sound natural either.
Why don't I get that strong feeling of command in said to him like I would in direct imperative speech i.e, "Buy Milk"