The book is correct.
As I understand your question, you wonder if
Peter was seen to drive away
Would be understood to be using the idiom "see to it" meaning "make sure something is done.
That would not be possible. I could say "I saw to the drive" to mean "I made sure the drive was ready" (Odd but possible). I could even say "The drive was seen to by me" (very odd but still possible). Here "drive" is a noun meaning "a short road leading to a building"
However it is not possible to say *"I saw to Peter drive away". It cannot take two objects and it cannot take a bare infinitive as an object. So this active sentence is not correct. As such you cannot understand "Peter was seen to drive away" as being the phrase "see to"
Instead, it is the simple verb "see". The book is correct "Peter was seen to drive away (by me)" is equivalent to "I saw Peter drive away". The active verb "see" can have a bare infinitive (drive away), but the passive form needs a to-infinitive "to drive away".
The active verb "see" must have a bare infinitive "drive away". The passive must have a to infinitive. So we don't say "I saw peter to drive away"; that is not good English. It does not mean "escorted"
The sense of escorted could be written "I saw to it that Peter drove away". I suppose "That Peter drove away was seen to" would be a passive form. No one would actually use the passive for this.
Your suggestion of using a gerund is (in this case) equivalent. "I saw Peter driving away" or "Peter was seen driving away" have the same meaning as the examples that use the infinitive