I've only heard that phrase in passing a few times, but from context it appears to mean "social outliers". Meaning, people who are culturally different from any other group that is included in the study and have no real cultural connection with the group as a whole.
This would fit pretty well with how I remember the phrase being used, and it makes sense in terms of the phrase itself.
To explain the phrase: We create scales to measure weight or, more generally, to measure things. But, in creating a scale we assume several things about what we are measuring. For example, we may assume that the object we are weighing is going to be a heavy cube of some metal. So we build a scale with a plate for us to put the metal cube on, and that works fine for heavy cubes. But what about a metal ball? It would roll right off! Or perhaps a helium balloon? That's just going to go up! Both of those objects "do not fit the scale." They are outside of our understanding of how to weigh things.
Hence, that's what the sentence is saying about some cultures. They exist outside of the other culture's expectations and assumptions. Thus the other cultures have no framework or understanding to view them through. They're like balloons on our scale, they don't fit our expectations, and thus we can't measure them properly.