While reading an article I came across the following sentence:
The more successful group, in terms of their creativity quality of their solutions...
Is this grammatically correct? Can a noun be used to describe another noun?
A noun can act as an adjective to another. If it does so, it's called an attributive noun (or a noun adjunct).
However, the specific sentence in the question is ungrammatical.
✘ The more successful group, in terms of their creativity quality of their solutions . . .
The following would both be grammatical versions of the same sentence:
✔ The more successful group, in terms of the creative quality of their solutions, was . . .
✔ The more successful group, in terms of their creative quality, was . . .
Which one to choose would depend on what was actually being expressed. (Since it's not a complete sentence, it's not entirely clear.)
Also note that, stylistically, some people might choose to use its as the pronoun for a group rather than their.