I think the hyphens are necessary. It is hard to read otherwise. The hyphens make it clear that "not sufficient enough" is a single modifier that applies to "data." Otherwise, it looks like you are saying "The algorithm suffers from not sufficient enough data," and then when I get to "modelling techniques," I wouldn't even know how that fits into the sentence.
But I should point out that this sort of three-word modifier is informal no matter how you write it. Also, "sufficient" and "enough" are roughly synonymous, and putting them together this way is sort of unclear and redundant.
If you are writing for something like a formal essay, it would be better to write
That algorithm suffers from insufficient data modelling techniques.
That algorithm suffers from data modelling techniques that are not sufficient.