With the catenative verb "try", the meaning depends on the kind of subordinate clause functioning as its complement. For example:
 You should try to eat less.
 You should try eating less.
In  "try" means "endeavour", while in  it means "test the effectiveness of".
Now compare your examples:
 I tried to order a book on line but I did not succeed.
 I tried ordering a book on line but I did not succeed.
 is fine where, like in , "tried" means "endeavoured". But  is wrong since it makes little sense to say that your testing the effectiveness of ordering a book on line did not succeed.
Edited for accuracy and improved clarity.