In the phrase “machine learning”it is really confusing for me to get the meaning of it in its grammatical sense.
Assuming that you're talking about the kind of machine learning that they study in computer science, the word machine in your example is a noun that's used as an adjective (it's more formally known as a noun adjunct in the world of linguistics) that modifies (or describes) the main noun, learning, which here refers to the processes by which a computer system is capable of acquiring knowledge without being explicitly told to do so.
Yes, the word learning can be a noun and is a noun in this case. The whole thing would be called a compound noun because it's made up of two separate words—a noun and a noun adjunct. The main word in a compound noun is always a noun (I honestly can't think of a case where it's not). That's why learning in regard to your example can't be an adjective or gerund. Here's a quote from the Wikipedia article on compound nouns in English to support my assertion:
Most English compound nouns are noun phrases (i.e. nominal phrases) that include a noun modified by adjectives or noun adjuncts.
To put it simply, machine learning is the name of a filed of study and identifying things by their names is what nouns and compound nouns are used for. At the end of the day, it doesn't even matter what part of speech learning in machine learning really is. It's irrelevant since it's just a part of a bigger whole used to refer to something by name.