Both sentences are perfectly grammatical, and not wrong. "Brainstorming ideas" seems a bit redundant, because anything you brainstorm will be an idea. One could say:
I am brainstorming plans for a mathematics paper.
Simply "in" or "for" mathematics is very broad, and doesn't tell the reader/listener much. What kind of mathematics, or what task in mathematics, are you attempting to think of ideas for? But i don't think "brainstorm X in Y" is always wrong.
I am brainstorming new techniques in marketing.
sounds perfectly acceptable.
But "brainstorm" itself has become something of a cliche and in my view is overused. One should not say "brainstorming" when "trying to think of" carries the meaning equally well, i think. There are specific brainstorming methods, particularly for small-group use, in which one proposes many ideas with little or no refinement or critical analysis, and then the group winnows them, rejecting ones that seem impractical and refining ones that are retained. If you mean to indicate that you are using such a method, "brainstorming" is the perfect term. But for just trying to devise an idea or plan, simply 'thinking' may be better.