If you think about the art-related definition of "sketch," you could say it's applicable to a drawing that's pretty well done, but not quite right. Maybe it needs a little refinement because for some reason or another, the overall picture or maybe just a certain part of it just seems a bit off. (Maybe things are out of proportion or the perspective isn't quite right or whatever.)
Then imagine, say, a business that seems a bit off. Maybe they do legitimate business in the front but trade in something illegal in the back. You might walk into the business not knowing that they do illegal business in the back but just get a feeling that something isn't quite right. Something's a bit off – like the picture. So you could call the place "sketchy" (or also sometimes just "sketch" – still an adjective, not a noun).
Another thing commonly described as sketchy are neighborhoods. Maybe there's an up-and-coming neighborhood that seems pretty nice and has some good residents, but at night time some more unseemly characters start lurking around. Some people might call the neighborhood "rough around the edges," which is an idiom in English that when interpreted literally brings to mind a sketch because the lines in a sketch are often not totally crisp – they might be a little jagged or rough from dragging the pencil back and forth while perfecting the shape you're working on (e.g., in trying to make a perfect circle).
It's with the kind of imagery that's applicable to businesses or neighborhoods in mind that "sketchy" likely became more popular and eventually grew to be more of a generic term that could be applied to a lot of different things, including people ("frat bros," in your example sentence), where the relationship between the slang meaning and the original, literal meaning maybe isn't so direct.