is an idiom meaning that it happens more than a very few times
You might say
I smoked marijuana two or three times back in college.
If that were true, you would give a misleading impression were you to say
I took drugs back in college.
So the plural form “drugs” at least references a multitude of uses of at least one type of illegal drug.
I smoked marijuana often back in college but no other illegal types of drug.
If that were true, it might give the wrong impression to say
I took drugs often back in college.
The most idiomatic use seems to imply multiple times of use with more than one type of illegal drug.