In Japanese, referring to people as "it" considered rude. Instead of "it", We say "that person" or information of the person (ex. a person who wares a blue shirt).
In English, Is referring to people as "it" considered rude?
English doesn't have the same kind of strictures regarding how to reference people as Japanese does. We don't, out of politeness, refer to people as "that side" or "next door" (the way my Japanese in-laws do), but we definitely don't refer to people as it unless we are being extremely rude or condescending. Even then, this is something that would be said only in the heat of anger, and would be hard to take back.
Even babies are called he or she if the gender is known.
At all costs, avoid referring to someone as it.
In English, you use it to refer to objects, not people. So it would be very rude and not grammatical - not only do you use a wrong pronoun gender, you also implicitly objectify that person.
You should use he or she for men and women respectively. If you're looking for a gender-neutral pronoun, they is a commonly accepted option - although it's also slightly impersonal and might offend some people. The debate on proper gender-neutral pronouns is rather long and heated, though, so better not to get into it too much.
Sometimes you'll see it used to refer to very small children - infant and young toddlers - but it's still best to avoid it in favor of gendered or proper gender-neutral pronouns.
EDIT: The comments raise a good point on gender-neutral pronouns, so it might be worth it to expand on the issue.
Back in the past, if you didn't know the person's gender, you would simply default to "he":
?Every employee has to be at his desk at 1PM.
Today, however, this is considered rather rude - it implies that the male form is the default one, and the female one is an exception. That's probably not an implication you want to make. You might, however, still see it in the wild, along with the (these days seemingly a bit more common) generic "she", so it's worth it to know that it might refer to both genders depending on the context.
A somewhat better alternative is to use "he or she" (or he/she, or s/he, etc.):
?Every employee has to be at his or her desk at 1PM.
While it is inclusive towards men and women, it doesn't include people who wouldn't want to be referred to as either - people with androgynous, genderfluid, or otherwise non-binary gender identity. It's also awkward to use when referring to a specific person:
*Charlie has to be at his or her desk at 1PM.
It's usually suitable, but if you don't know whether all the recipients of your message fit into either of the two categories, it's better to stray on the safe side.
Singular "they" is currently the most common gender-neutral pronoun, and in most situations it's an acceptable solution:
?Every employee has to be at their desk at 1PM.
That doesn't mean it's free from critique, though - some people perceive it as ungrammatical (due to its association with plural forms), others consider it impersonal and objectifying. In some cases it also doesn't sound quite right:
?/*Either the husband or the wife will have to sign it themselves.
There have been attempts to find a better alternative with dedicated gender-neutral pronouns - Spivak pronouns, zhe, xe, and several others.
?Every employee has to be at zhers desk at 1PM.
Those, however, have not entered common usage for now, and are mostly associated with LGBTQ communities. They're also criticized as "unnatural-sounding" or "artificial".
Currently, using the singular they seems to be the best option, at least until a better solution catches on. It depends on your audience, though - as I've said, there's a lot of debate on proper use of pronouns, especially regarding the last category.
It is OK for babies.
If the baby's gender is unknown to you, you can call a baby 'it'. The parents will then kindly inform you on its gender "it's a boy (or a girl)".
Beside that, it is rude to use "it".
I have referred to people as 'it' in the past, but only when very annoyed with them. Usually it helps if there is a third person there, so you can turn to that person and say 'look at it', 'look at what it's doing'. But yeah, not a nice thing to do.