I think there's a few things going on here...
Whom's that book by?
The contraction "whom's" sounds odd, because "whom" is now generally used in formal contexts, and contracted "is" in informal contexts.
Whom is that book by?
Next, the word order makes it hard to see which is the subject and object in the sentence. If we switch that around, it becomes clearer that the subject of "is" is "that book":
That book is by whom?
As Teaspoon's answer points out, you can usually substitute "he" for "who" and "him" for "whom". In this case we could answer the question like this:
That book is by him.
This suggests that "whom" is indeed correct. But what about the word order? Let's compare a different phrasing:
The book is by which person?
The book is by that person.
The question is understandable, but doesn't sound natural, because we generally rearrange the word order to make a question:
Which person is the book by?
The book is by that person.
(Some grammarians would object that the preposition should not end the sentence, but there is little evidence that natural English has ever followed such a rule.)
Put together, that suggests that the technically correct form is:
Whom is that book by?
However, as others have noted, the use of "whom" in English is declining, so in an informal context, as implied by the contracted form, it would probably be:
Who's that book by?