With the catenative verb "try", the meaning depends on the kind of subordinate clause functioning as its complement. For example:
[1] You should try to eat less.
[2] You should try eating less.
In [1] "try" means "endeavour", while in [2] it means "test the effectiveness of".
Now compare your examples:
[3] I tried to order a book on line but I did not succeed.
[4] I tried ordering a book on line but I did not succeed.
[3] is fine where, like in [1], "tried" means "endeavoured". But [4] is wrong since it makes little sense to say that your testing the effectiveness of ordering a book on line did not succeed.
Edited for accuracy and improved clarity.
Verb
-ing is fine, but try toVerb
has a Gricean implicature of failure.