The 'formal' here need to be understood mathematically. (So are many other terms in a math book.) It means the 'math science' is about axioms and deductions using axioms and certain deduction rule(s), that is, no real-life means of gaining knowledge is taken into consideration when doing a math problem. That said, the nearest definition given by merriam-webster is:
relating to or involving the outward form, structure, relationships, or arrangement of elements rather than content.
As you can see in Hilbert's Foundations of Geometry, which is one of the most famous books written in the spirit of formalisation among mathematicians, every theorem is gained through rigorous logic deduction. Here, the method of drawing a picture to see something intuitively is not considered a valid proof[1]. However, methods like drawing a picture are widely used in other science or technology areas to help you get some result or prove something, which is one of the traits that differentiate 'formal' from 'not formal'.
Having some knowledge of formalism, mathematical logic and formal system may help you better understand the connotation of 'formal' in math. By the way, I feel like this question is more about philosophy of mathematics than English.
footnote:
[1] You may see there are some pictures in that book, but they are intended to help understanding rather than serve as a proof. In a research level math book, all pictures of geometric shapes, which are rare if not nonexist, can be removed without any reduce of the validity of the proofs there.