I've been told that in American English, sometimes words ending in -ough are written -u: for example thru instead of through.
Is this correct English, or is it simply a common error?
If it is correct, what are the rules for this spelling?
"Thru" is correct (however very informal, not a very good idea, and only used when space is at a real premium — e.g. road signs, technical drawings) English, but -u is not a shortened way of -ough except in words that derivate from through (e.g. breakthrough).
From memory, I can recall although, enough etc. where -ough can't be replaced by -u (althu, enu etc.), since in those words -ough doesn't have a /u/ sound. (However, although can be shortened to altho, as noted in a comment — however, Wiktionary and other dictionaries register it, noting that it's quite informal.)
As a speaker of American English, I would understand what was meant by thru for through, but I would find it very informal. The Oxford Dictionary entry supports this, saying that it is:
chiefly North American; informal spelling of through.
The rule, however, isn't that words ending in -ough are shortened to -u, but instead in very informal writing the last vowel sound of a word is used instead of the proper ending. This gives through → thru because of the final "oo" sound, but also abbreviated forms though → tho or although → altho because they end in "oh" (as in "cold").
Shortenings you are more or less likely to see for "-ough" (most are, as previously noted, considered at best very informal and at worst a severe case of bad spelling):
A couple of "ough" words are never shortened due to either the word's low frequency, variability of pronunciation, or to any replacement being a collision with another word: sough, dough, brough, bough, trough.
Thru is a very informal spelling that is never used in print or school. Through is probably not the best way to spell it in textspeak, because textspeak avoids long traditional spellings.
But in formal and semi-formal (anything but the most informal) you should avoid using such simplifications unless you are in a place where you've seen others use it. For example, if you use thru here at ELL, it won't be considered a misspelling but will definitely be considered too informal.