What is the role of "by studying in groups" in the following sentence? Is it ok Grammatically?
By studying in groups, students can learn better.
This sentence is grammatically correct. It can also be written as "Students can learn better by studying in groups," which doesn't change the meaning, but can change the emphasis.
"By studying in groups" is a prepositional phrase, which modifies (or expands upon) another phrase or clause in the sentence. A prepositional phrase is normally a subordinate clause, which mean that it cannot stand on its own as a sentence*.
In your example, "by studying in groups" is modifying the predicate "can study better", by specifying the methodology of how this can be done.
*In informal usage, native speakers sometimes break this rule, such as when answering a question, by omitting portions of our response that are a restatement of the original question. For example, the question "Where is my cup?" might be responded to with the prepositional phrase "On the table", rather than the full sentence "Your cup is on the table".