A good way to check this is with an etymology dictionary. For example:
rudiment (n): 1540s, from Middle French rudiment (16c.) or directly from Latin rudimentum "early training, first experience, beginning, first principle," from rudis "unlearned, untrained" (see rude).
So rudimentary knowledge is something taught to someone who is completely unskilled. For example, a rudimentary carpentry skill might be how to hammer a nail in straight (without hitting your thumb), or how to glue together two pieces of wood.
Both "fundamental" and "foundation" share the same Latin root fundare ("to lay a bottom or foundation") so both words mean much the same thing. In contrast with rudiment, foundational/fundamental knowledge forms the basis for everything else in that field. Again, using carpentry as an example, this might be how to select good wood for a particular project, how to create proper joins, how to properly build load-bearing structures, how to use and maintain your tools, and so on.
Obviously there is some overlap between rudimentary and fundamental, since you need to teach the basics to someone untrained. But the intent is different -- it sounds much better to say you have a strong foundation in a discipline, rather than saying you have picked up the rudiments of that discipline.