Although the word 'dude' is not disrespectful, why do some people use it only when they're angry? Could it be that the word is insulting?
The word "dude" was originally a mocking term around 1890 in the US for a city man who wore overly fancy clothes. In the West, it became a term for city people who were out of place in a frontier environment. For example, a "dude ranch" still refers to a place where men from the city come for a vacation and pretend to be cowboys. Then, finally, it became a slangy way for young people to address each other. This final change probably happened first in African-American culture in New York and then got picked up by surfers on the west coast. People now think of it as a stereotypical expression for Californians or surfers. Because of this history, the word has always been informal, and it has never been an especially respectful way to address someone who is older or in a position of authority. Compared to similar words like "man," "brother," "bro," or "buddy," it has a different shade of meaning which is that it often implies pretty aggressively that the person being addressed is socially equal to the person speaking. (This makes sense because of its connection to surf culture.)
A man is standing in the parking lot outside the supermarket and asks me, "Brother, can you spare some change?" This is an extremely humble way for him to approach me, and it gives the impression that he's trying to placate me. If you substitute "buddy," the meaning is about the same. But if he uses "dude," the effect is completely different. It gives the impression that he specifically wants to avoid sounding like a humble beggar.
I see a policeman in an angry dispute with a man on the street. The cop pulls out his club and waves it at the man. I'm standing nearby, and I say to the cop, "Brother, let's all be peaceful here." This comes off as just what it sounds like, an appeal for peace. But if I say, "Dude, let's all be peaceful here," the implication is that I'm saying we're all equals, his uniform doesn't give him legitimate authority, and I'm criticizing him. If I say this, I might end up in handcuffs or with my teeth knocked out.
I'm a college teacher. I'm teaching a lab class, and I have some french fries. One of my students says to me, "Dude, could I have one of those fries?" This is friendly, and OK in California culture. But if the student comes to my office hours to complain about his exam grade and says, "Dude your course is ridiculously hard," then he's being very disrespectful.
As you can see from these examples, the shades of meaning in this word are very subtle and dangerous, and they depend on details of culture and context. If you're not a native English speaker and not culturally fluent, then you would be safest using "dude" only with people your own age and status, and not in a context where you're complaining or criticizing.
I’m guessing you are referring to the usage as an interjection/exclamation when one is angry or annoyed.
5 (an expression of shock, approval, sympathy, or other strong feeling):
Dude! That's one expensive sandwich!
Notice that it can be used to express a variety of “strong feelings”. The usage in the entry seems to express shock or surprise. An example used in anger or annoyance would be
Dude! What the hell! Why did you eat my sandwich? I was saving that for dinner!
In other words, the word itself is not so much insulting. Rather, it can be used to express a wide range of emotions. It seems the usage that you (OP) are most familiar with is the usage expressing anger.
I generally agree with the two answers that have been submitted so far, but would like to add some nuance.
"Dude" is an informal way of addressing someone. You would usually use it only for people you know well and are close friends with. Although there are times when a stranger may call you "Dude" (or vice versa) as a way of showing non-malice and emphasizing informality. Calling someone "Dude" can even carry a tinge of humor because it imitates stoner, sufer, and skater culture. (See definition #2 at https://www.urbandictionary.com/define.php?term=dude.)
This is a little off topic, but why is stoner, sufer, and skater culture seen as humorous? Probably because of numerous, exaggerated characters in TV and film that are oblivious to social norms and always act and speak informally. For example, see the character of Jeff Spicoli in Fast Times at Ridgemont High, or the two lead characters in Dude, Where's My Car?.
In that light, beginning a complaint with "Dude" is actually not disrespectful or insulting at all. It's a way of appropriately showing anger, while at the same time showing that you haven't lost your sense of humor and that you still feel generally friendly towards someone. You're just expressing how you feel and showing a willingness to accept their apology or to listen to their side of the story. It can defuse some of the tension that comes with criticism.
Of course, because "Dude" is informal, it could be taken as disrespectful or insulting when used in a situation where formality is expected.
It very much depends on your cultural context. In the rural western US, it is mildly insulting/contemptuous, describing a person from more urbanized places who really doesn't know how to behave in the west. See for instance "dude ranch", or compare to the more general use of "(adjective) tourist".
It could also be applied to someone who is dressed fancily, e.g. "He's all duded up, must be going to see his girlfriend".
In more urban cultures, it's apparently become a general form of address to people (mostly men) with whom the speaker is on somewhat familiar terms, e.g. "Hey, dude, look at that!" Such use might be considered mildly insulting, but (IMHO, at least) not inherently. Rather, it's because it assumes an unwarranted degree of familiarity on the part of the speaker.
It might have still other meanings in other subcultures, but I'm not familiar enough with them to say.
It's context dependent, as well as somewhat culturally dependent.
Historically, it was a mildly derogatory term roughly equivalent to 'dandy' (that is, a man who is somewhat obsessive about his clothing and appearance, often implying that he looks far fancier than he really is), though over time this has shifted somewhat significantly.
Modern definitions include that original historical definition (though it's usage this way is relatively rare at least in the US), as well as the slang definitions pointed out by many other answers (either an informal generic for a man, or a rough equivalent to 'buddy', 'chum', or 'mate' (in the Australian English sense of 'mate', not the biological sense)), as well as occasionally a term for either someone who grew up in a big city (equivalent to 'city slicker', kind of implies that they're ignorant of the country lifestyle, generally mildly derogatory) or in the Western US, an Easterner who is vacationing on a ranch (often implied to be pretending to be a cowboy, this is the sense that the 'raised in a city' meaning came from and was more prevalent historically than it is today). It's also used as an interjection to express surprise, amazement, or being impressed by something.
The important thing here though is that it's invariably informal, and when used to refer to a person almost always implies either roughly equivalent social standing or a high degree of familiarity (or both). Because of this, in formal contexts it's likely to be considered at least mildly offensive and possibly insulting to the person being referred to, but in informal contexts it's usually not an insult, but could be interpreted as one by people who are especially sensitive (usually about their gender identity, as 'dude' still carries a heavily masculine connotation).
In some cases school age children (mostly boys) may also try to use it as an insult, usually referencing definitions that I can find no other attestation to typically involving unflattering comparisons to infected body parts of various large mammals (which is funny, because there are plenty of words that actually do have etymologies that would make them rather insulting in a similar manner if interpreted historically), but in actual serious conversation you're not likely to ever encounter this type of puerile insult.
Dude - okay
Dude! - okay
DuuuUUUUde - okay
duuuUUUUDE - okay
DUUuude!! - hostile/insulting/upset
We native speakers don't usually think English is a tonal language. We usually think things like Chinese and Thai are tonal languages, but that English is not. However, this is not true.
Languages like Thai and Chinese use tones to pronounce words and possibly phrases. On the other hand, languages like Japanese and English use tones to pronounce context. And it is a big deal in English.
If you're using English, be careful when you write emails or text messages, instead of speaking - emails and texts are sometimes misunderstood, because there is no tone! Using emoji and very casual, rule-breaking, non-textbook English and Japanese helps in cases like this. You can use emoji and non-textbook English/Japanese to help show tone. And when speaking Japanese, body language is really important too. (Tone and body language are important in both languages, but English mostly uses tone, while Japanese is maybe more 50/50.)
You are asking about the word "dude". In these situations, tone is the reason it is hostile or insulting. The word "dude" is perfectly okay, and people use it often in casual English in many situations. However, the tone you use to say "dude" changes the context.
"Dude" is slightly special though: On the one hand, the word is perfectly okay and normal. On the other hand, it is still popular to use that specific word with a hostile or insulting tone - That means the same thing as "What the heck!?"
As you say, some people seem to only use "dude" in the hostile/insulting way. But that is just them, not the general situation.
"Dude" is sometimes said to be an aggravated hair on a large mammal. This is rather esoteric knowledge and does not factor into common parlance.
Dude is in no way derogatory or disrespectful, but is not appropriate in formal speech. It has a counter-culture vibe as it was popularized in 1960's surfer culture. More modern culture in the 1990s say use of dude by Bart of the Simpsons and in "Bill and Ted's Excellent Adventure".
FYI: Dudette is the female equivalent, although "dudes" can be used to address mixed-sex audience. "Dudes" as in the all-inclusive "you guys"* or be separated as "Dudes and Dudettes" as in "ladies and gentlemen".
- There are some who would say that "you guys" is gendered speech. That may be true and no determination is made here. However the vast majority of native english speakers are not offended when "you guys" is used to address a mixed audience.
In addition to the other answers (and I definitely agree with the aforementioned connotations of slang-level, male-gendered, and presumed familiarity) saying a name for someone, whether it's "Lady", "Ma'am", "Dude", "Mr", "Brian" (or whatever the person is called), "Man", "Miss" etc can be a way to rudely say "Hey, you! Listen! Right now!" You're not listening to them but you're trying to grab their attention. (Sometimes for good reason, sometimes not, but it can still be jarring.)
– "So I was wa…"
– "Dude! Check out this gnarly scar I got the other day!"
The first person can get miffed because they wanted to finish telling their story.
"Dude" is tricky. It depends on the situation, the social position of the participants, the tone (very important), and the personalities involved, so it can be interpreted as anything from very friendly to hostile and insulting. The rules are complex to non-existing.
As an English learner, you should probably avoid using it, because it is too easy to get wrong.