I completely agree that the present should be used in your first example since both of the sentences, as they are now, sound just a bit off.
With that being said, we cannot deny the fact the latter is correct as it collocates with the action of "trying" and might be able to "work out", while the former makes the meaning empathic since we are talking about a terminated action of " working out", and using the present perfect.
The following could be synonymous with your second example.
I've tried to order the book, but it has yet to work out that way.
(Perhaps the store or website isn't able to provide you with that
service now, but it may be able to at some point of time/soon);
I've tried to order the book, but it doesn't work out that way. (
expresses a clear statement )
If you really want to use the past simple in your sentence, it's recommended to omit the perfect tense as it isn't necessarily used there. But its usage is clear and correct.
I [have - additional emphasis on the time/action of "trying"] tried to order the book, but it didn't work out [a terminated period
in the past, no longer taking place, nor does it give any time
consideration at this moment] that way.