Neither construction is correct.
more number of tasks
The problem with this phrase is that number, when used in the sense of 'quantity' (ie. "a quantity of Xs"/"a number of Xs"), is a mass noun whereas more only works for count nouns. Your other suggestion almost fixes this:
more numbers of tasks
Here numbers could be taken as a plural count noun, and so could be modified by more. The problem is that it no longer has the meaning of 'quantity', and simply refers to numbers as things (eg. "Which of the following numbers are even?"). As such, it doesn't make sense to say "numbers of tasks".
To confuse things, it is possible to use numbers as a mass noun as well, for example: "Significant numbers of voters are turning away from bipartisan politics." However, it's not so appropriate in this context, and in any case would not admit use of more.
If you insist on using number, the correct construction would be:
a greater number of tasks
Here, greater is the appropriate modifier for a mass noun. I'm not sure what you mean that it should be number because you're talking about equations. Since the word is being used in the sense of 'quantity' rather than to refer to mathematical numbers, there's no necessity to use it, unless you want to achieve a kind of punning or poetic allusion to the fact that numbers are involved. Otherwise, it would be much more natural to say simply: