There is no simple rule. But here is the key: in English, you must learn to recognize phrases, not just individual words. In English, phrases are often indivisible units of grammar.
In your example, "bright red" is an adjectival phrase. The two words together function as a single adjective. You think of them as one unit:
She wore a bright red beautiful dress.
If you re-ordered it, you would say:
She wore a beautiful bright red dress.
If you re-ordered it like this:
She wore a bright beautiful red dress.
that would change the meaning, since "bright" would now be an adjective modifying "beautiful red dress".
A phrasal adverb
Here is a phrase that functions as an adverb:
Paul quit smoking cold turkey.
"Cold turkey" is an adverb meaning "to quit suddenly, not gradually". How do you know? The same way you know the meaning of any individual word. "Cold turkey" is itself an element of English vocabulary, even though it's two words long.
The adverbial phrase "cold turkey" is mainly used to describe quitting smoking. In a different context, the same two words would be interpreted as an adjective modifying a noun:
There is some cold turkey in the refrigerator.
Here, "turkey" is the simple subject of the sentence, modified by "cold".
This doesn't mean that you couldn't use "cold turkey" as an adverb to describe quitting something other than smoking. But if you say something like "I'm giving up potato chips cold turkey", people will understand that you are drawing an analogy with quitting smoking. It wouldn't make sense to say "I'm quitting my job cold turkey" because there is no analogy between quitting a job and quitting smoking. Then again, if, for some strange reason, your job was a bad habit or perhaps caused lung damage, and it was possible to gradually reduce your number of working hours, then "cold turkey" would make sense in that sentence.
Are you terrified yet?
Now that you know that there is no systematic way to tell an adjective from an adverb, because phrases in English can function like words and there are no grammatical cues in the sentence to tell you when that's happening, is learning English starting to seem hopelessly difficult? Well, the situation is even worse than what I've told you so far. Here are three more difficulties.
First, the phrases are not necessarily contiguous in a sentence. Look at this:
I'll pick you and your friends up from the airport.
"Pick up" is a phrasal verb. If you're not used to this, you can easily think that the verb in this sentence is "pick". But "pick" means something completely different than "pick up", "pick out", "pick apart", "pick over", etc. You have to train your ear to recognize the phrase even when it's interrupted by other words. You have to be able to do that even though "pick" can be a verb on its own, as in "I'll pick a new flight for you and your friends."
Second, the indivisible phrases are not completely indivisible. It's still possible to modify individual words within them to modify the meaning of the phrase, though this is somewhat uncommon. For example, a person could say "Well, I'm sort of quitting warm turkey. My last cigarette is scheduled for Sunday." (By the way, "sort of" is another phrasal adverb.) What's happening is that people simultaneously perceive the phrase as an indivisible unit and as individual words modifying each other ("turkey": noun; "cold": adjective). Inside "bright red", "red" is perceived as a noun, modified by the adjective "bright"—even though "bright red" is itself an adjective.* That's why "bright pink", "bright yellow-green", etc., also make phrasal adjectives, and not "bright pinkish", etc.
Third, English pedagogy rarely acknowledges the way English grammar depends on phrases and analogies. For centuries, the teaching of English grammar has focused on rules, which were thought to govern individual words and how they combine to form sentences. This originated when English grammarians got their terminology and concepts from Latin, which is genuinely word- and rule-oriented. More recent work by linguists has started to take the phrasal orientation of English more seriously, but it's still usually explained as a system of strict rules rather than the analogy-oriented way that it works in real life; and few teachers have even heard of these new systems for describing English grammar.
So, most teachers, especially teachers who are not native speakers of English, will mislead you about English grammar. They were "taught" that it's all rules, operating on individual words, and they'll pass that misinformation on to you. And many native speakers, having been mistaught English grammar in elementary school, will "helpfully" try to answer questions about English by telling you word-oriented rules, never saying a word about phrasal units or the role of loose analogy in combining and varying them.
Reason for hope
OK, now you know the worst of it. Now here are a few reasons why you can expect to eventually master the "unruly" phrasal grammar of English.
First, English also has a word-oriented, rule-oriented grammar. In "She wore a red dress", there are no indivisible phrases: "red" is an adjective modifying the noun "dress". This is the backbone of the grammar. The phrasal stuff combines and modifies the basic elements. After you've mastered the basic grammar, you start to perceive phrasal units like "bright red", "cold turkey", and "sort of".
Second, now you know about phrasal units in English. That's a huge advantage. Now you know to learn "pick up" as a phrasal verb. Now you know not to think that "up" in "pick up" introduces a prepositional phrase—a common mistake for people whose first language is a Romance language. Just knowing that these phrases exist helps you "tune your ear" to hear them.
Third, the potential for chaos is limited by the fact that people really do communicate in English. While unruly phrasal grammar could theoretically make the language hopelessly irregular and complex, the use of indivisible phrasal units is kept in check by common patterns and customs. With experience, you'll pick up those patterns and customs intuitively, just like a native. It's mostly hard at first, before you've mastered the basic grammar and before you know that these phrases exist.
Also, while most of the world's languages do not have a phrasal grammar like English, some do. If you're starting from another language with this kind of grammar, like Mandarin, this aspect of English won't be an obstacle for you. (In fact, Mandarin grammar is much more phrasal than English grammar.)
*Actually, "bright red" is only an adjective if you use it as an adjective, as in your example sentence. "Bright red" can also be an ordinary noun phrase, with "red" as the noun, if you use it that way, as in "Bright red is my favorite color." "Bright red" as an indivisible adjectival phrase is formed by analogy with "bright red" as an ordinary noun phrase, drawing on the fact that "red" ordinarily serves as both an adjective and a noun.