My own instinct would have been to remove them.
I think, the above sentence is in passive construction. In passive construction, we need to use past participle form verb. And in the above sentence, been is the past participle of verb, and if we want to use another verb after past participle of verb in passive construction, we will have to use to-infinitive. Am I right?
This is not a passive sentence, but an active sentence employing a past-tense modal perfect.
The main verb, the final verb in the group is BE. Note that the verb BE cannot be cast in the passive, because it is intransitive: it does not take a Direct Object.
If we rewrote your sentence in the simple present it would read:
My instinct is to remove them.
BE takes the past participle form been because it is the complement of HAVE in the perfect construction. In the present perfect this sentence would read:
My instinct has been to remove them.
HAVE takes the infinitive form have because it is the complement of would. Modal verbs always require the following verb to take the "unmarked" infinitive form—that is, an infinitive without the marker to.
My instinct would have been to remove them.
In the phrase to remove them the verb REMOVE is cast as a marked infinitive, to remove, because that is how we express the complement of the noun instinct: guard dogs have an instinct to attack intruders, mothers have an instinct to protect their children. This complement is also employed for the Predicative Complement of a linking verb when instinct is the Subject:
A mother's instinct is to protect her children.
My instinct is to remove them. (see my first 'rewrite' above)
