Non-satisfaction to express that a constraint is not satisfied is fine, though I'd prefer it hyphenated. Note that in everyday English, the noun to express that someone is not satisfied with something is dissatisfaction, and the corresponding adjective is dissatisfied, but these words are never used in the mathematical sense: a constraint can only be unsatisfied (which also exists in everyday English, but with a different meaning) or not satisfied.
However, I don't understand the sentence. (Non-)satisfaction is boolean: it's either true or false. This isn't something that can be exceeded. What causes the cluster to be discarded? Is the non-satisfaction of the circle constraint related to the orientation? There is a similar problem with the maximum length: what does it mean for a length to exceed an orientation? Even the notion of an orientation exceeding a defined orientation is strange, but maybe that's because of my lack of familiarity with algorithmic geometry: I perceive orientation as something that can go round in a circle (or more in higher dimensions), not something that has an order relation.
Here is one way to write this sentence which is grammatical and comprehensible. Beware that I don't know if it expresses the right mathematical property, because I don't know what mathematical property you're trying to express.
In contrast, if the orientation or the maximum length of a rectangle exceed the allowed range, or if the circle constraint is not satisfied, then the cluster is discarded.