Suppose this sentence
Otherwise, if it has an ancestor which encloses the target section, another anchor within the section which shares the same ancestor is required;
Can I say it as
Otherwise, if it has an ancestor which encloses the target section, another anchor within the section is required which shares the same ancestor
Specially in passive sentences?
Passive or active is irrelevant.
In parsing a relative's antecedent, the reader tracks back to the first entity capable of sustaining a relative. Consequently, putting the relative anywhere except immediately after its referent makes parsing more difficult (which is a discourtesy to your reader) and makes misparsing more likely (which is damaging to your argument).
In your second example, the reader tracks back first to the entire clause another ... is required; it will take her a couple of beats to realize that a requirement is unlikely to share an ancestor. She then tracks back to section, and takes you to mean that the section must share an ancestor. She may never arrive at the referent you intend, which you have told us elsewhere is another anchor.
A relative should be placed immediately after its referent. However, that yields another anchor which shares the same ancestor within same section; this is acceptable only if it is the ancestor, not the anchor, which must be within the section.
If you cannot place your relative immediately after its referent, you will have to rewrite to ensure that a misparsing cannot occur. For instance:
Otherwise, if it has an ancestor which encloses the target section, another anchor within the section is required, an anchor which shares the same ancestor.
This is still dubious, because I don't know what is meant by an ancestor 'enclosing' a section. If an element is ancestral to every anchor within a section it 'encloses', you have defined the ancestor twice.
