Basic structures:
For me these two examples are examples of the word TO used as an operator. Not an infinitive. The TO functions to reveal a purpose, result, outcome or opinion.
A)
This book is easy to read.
This question is hard to understand.
B)
My advice is to work more.
His idea is to sing loudly.
For me, these other two examples are infinitive complements, grammar-wise.
In sentences patterned like as the ones in A) above, you can add, basically, any adjective and precede the adjective with an entire range of adverbs.
This book is extremely complex to understand.
This question is very easy to solve.
When you have: subject + predicate + TO as an operator + a result or purpose in the form of a verb, any adverb can be placed in front of the adjective.
The adverb TOO is very common in English.
The sentence: /The table is too heavy to be lifted/ is not really grammatical. You could say: The table is too heavy to be lifted by us. However, it is not very idiomatic. Idiomatic is: The table is too heavy FOR US to lift.
What is grammatical is: /The table is too heavy (for us) to lift/
For the passive tense, you'd have to have something like this:
The lecture is too hard to be understood [by many students].
The speech is too long to be noted down by hand.
What determines A) is this:
Can you take: The book is easy to read and make it: To read a book?
The question is hard to understand: to understand a question.
So, the VERB after the TO OPERATOR has to make sense in this kind of transformation:
The bank is big to fail. To fail a bank: doubtful.
One does not "fail" a bank, though, conceivably, a regulator might fail a bank in an audit. To fail the bank [in the audit].
What has to examined is: whether the VERB can be made into a regular infinitival phrase with the noun. That's the question. The question is not about the adverb.
Compare the example above to:
Kids are hard to fail [to not support someone]. To fail kids. Yes, you can fail kids. Not be around to support them, for example. And also, the other meaning as to cause not to pass a test or exam: to fail kids.