... failure of the space M to be smooth is indeed grammatical English, but I'm whether it's the usual way to express what you want is a question for Mathematics SE.
Usually you see an abstract quality being measured, such as "smoothness" or "roughness":
- The function f measures the lack of smoothness of the space M
Here are a couple of examples from published papers:
... certain geometric measures of smoothness ... (https://www.jstor.org/stable/3212668)
It is well-known that random functions are typically “rough” (non-differentiable), which raise the question of determining and measuring roughness. Probably, the most studied roughness measures are the Hausdorff dimension and the p−variation index. (https://arxiv.org/pdf/0802.0489.pdf)
Multidimensional Whittaker graduations are obtained by minimizing a linear combination of a measure of lack-of-fit between the graduated and ungraduated values, and measures of lack-of-smoothness of the graduated values across each dimension. (https://www.sciencedirect.com/science/article/pii/0167668795000216)