what is the possessive plural of the French name Rioux, in an English text?
Is it the Riouxes' house
or the Riouxs' house?
You would form it based on the plural form of Rioux, so the first step is finding what that is.
According to the Daily Writing Tips article "Plurals of Proper Names" by Mark Nichol,
The plural forms of names ending in unpronounced -s or -x are identical to the singular form: “The era between the third and seventh Louis,” “The two Lacroix could not have been any different,” though “. . . Louis III and Louis VII” and “The two Lacroix brothers . . .” would be better.
So the plural of Rioux would just be Rioux.
To form the possessive of a plural noun that ends in the letter x, you just add an apostrophe.
So the possessive of the plural would be spelled Rioux', and your example would be "the Rioux' house."
However, this is not totally certain, because apparently Wikipedia said something different; see this answer from Reg Dwight on ELU: What is the correct possessive form of names ending in “x”?
In English the normal solution is periphrasis--which simply means you will avoid the problem by re-wording the statement.
Periphrasis isn't limited to English. It is a general strategy for avoiding uncertain declination or conjugation. It uses a different construction (in this case, something like "the house of the Rioux family") to sidestep the difficulty.
The alternative is not going to be something that will pass for correct English.