It used to be common for someone to refer to a person who speaks a little bit of English as someone who used or spoke "pidgin or broken English". I have also heard referred to newcomers who are obviously not from the country they are in as F.O.B, "fresh off the boat"