They are both fully grammatical and fully idiomatic, and refer to exactly the same objective circumstances.
The difference, as is usual in the case of a perfect form, is entirely in how the speaker is choosing to present the temporal structure of what they are saying.
In They hoped that he would finish his book by the end of the month, they are looking forward from when they were hoping.
In They hoped that he would have finished his book by the end of the month, they are imagining themselves looking backward from the end of the month.
In this case, there is very little difference in meaning; sometimes the choice sets up an expectation about the time of following sentences.
Edit: Books4languages has suggested an alternative interpreation of the second, as a counter-factual conditional (implying that he didn't finish by the end of the month). I agree that this is a possible reading, but it is not the meaning that occurred to me.