"Grammatical" and "idiomatic" are two adjectives that may well be used together, but together are not necessarily an "idiomatic phrase", so be careful with what you label as "idiomatic". You say you have found "lots of people" saying "grammatical and idiomatic", so that really should indicate that it is both grammatical, and idiomatic in the sense that it sounds natural, but it is not really what you could call a "commonly used" phrase.
Also, be careful with your use of ngrams. They can be useful, they can also massively misrepresent a matter, especially when you represent something that is a true idiom and is commonly said against something that is perfectly grammatical but may not be said as often. To use an analogy, that is like comparing apples and oranges.
Look at this ngram that measures "grammatical and idiomatic" against "idiomatic and grammatical". One way around is favoured more than the other, but both have been used. I would suggest that grammar is the first consideration because if something is not grammatically correct then it cannot be idiomatic.