The idiomatic standard is...
The hem came undone (about a dozen hits in Google Books)
Note that although it will obviously be understood, Google Books reports no hits at all for the highlighted 3-word sequence in...
The hem came unsewn (link)
It may be worth noting that if a hem "comes undone", that can only really mean the stitching has "unraveled" (hems don't usually have buttons). Whereas...
His shirt came undone (link)
...is more likely to mean the shirt buttons came undone, rather than the stitching. But obviously you can say My shirt stitching is coming undone.