In this position, the phrase "formed by tufts or loops . . ." could modify either "fabric" or "effect". "Fabric" is the closer word, and we should assume that this phrase modifies this word unless we have a reason to think that it doesn't. We have such a reason. In this sentence, it modifies "effect".
The reason that this phrase doesn't modify "fabric" is that it also contains "fabric". These aren't indeterminate tufts or loops. These are tufts or loops that stand up from the body of the fabric.
The idea of a fabric formed by a part of that fabric itself is confusing. The idea of an effect on a fabric formed by a part of that fabric is much more sensible.
The simplest way to indicate that the fabric consists of tufts and loops (rather than the effect) is to make the effect definite and to not mention the fabric again:
The surface effect on a fabric formed by tufts or loops of yarn is called pile.
Unfortunately, this version of the sentence does not mention that the characteristic tufts or loops extend from the fabric's body. That makes this version less truthful, or at least less precise. Many knitted fabrics have loops of yarn but do not have a pile.
The real problem here isn't a question of grammar. It's a question of logical consistency -- a question of meaning. The tufts or loops in question are not the entire fabric. They are just one part of it, specifically the part that forms that surface effect.
You may want to consider this version:
A surface effect on a fabric that has tufts or loops of yarn that stand up from the fabric's body is called pile.
Here, "that has tufts or loops . . ." modifies "fabric", without claiming that those tufts or loops are the entire fabric. We've replaced "formed by" with a phrasing that is still logically consistent and truthful.