(The following adopts the common convention of using slashes to denote phonemes and square brackets to denote phonetic realizations.)
Background
Japanese speakers' inability to distinguish /r/ and /l/ in English lies in the difference between the two languages' phonological systems. English distinguishes /l/, a consonant articulated with the air from the lungs obstructed by the tongue from going through the middle of the mouth and thus the air coming through the side(s) of the mouth (called lateral), and /r/, a consonant produced with the tongue raised close to the roof of the mouth but not so much to obstruct the airstream and with the tongue wide enough not to let the air through the sides of the mouth (called median).
English /r/ is pronounced in a variety of ways with very subtle differences, but here I use [ɹ] as a shorthand for a category of the most typical realizations, with the vocal cords vibrating (voiced) and the tongue not so close to the roof to cause friction (approximant). Consonants like [l] and [ɹ] are called liquid consonants.
Japanese, on the other hand, doesn't distinguish such sounds and has only one liquid phoneme, here notated as /ɾ/ to distinguish from the English /r/. /ɾ/ is most commonly articulated as a median tap consonant [ɾ], a brief movement of the tongue tip striking the surface behind the upper teeth (which is essentially the same sound as the one in the middle of butter, did it, etc. in American English—those who referred to the Disney song as レリゴー rerigō were spot on). But /ɾ/ is also realized as all sorts of sounds, including the lateral [l].
This means that replacing a sound of English /r/ with [l] in a word would not strike (untrained) Japanese speakers as meaningfully different, and they would perceive it as the same word.
So, in a way, a phoneme may be understood as a category of sounds in one's brain. English speakers are able to distinguish between [ɹ] (or something similar) and [l] (or something similar) because they have two categories (phonemes) for those types of sounds in their brains. Japanese speakers have only one category for liquids, so [ɹ], [l] and all the other soundalikes all sound the same to them.
Children start acquiring phonemes very early, before they reach their first birthday. Acquiring phonemes is like drawing lines in the sand. Infants are at first able to recognize very fine shades of sounds, but the number of sounds they are able to distinguish reduces as they are exposed to the language spoken around them and approach their first birthday. Where the future English speakers draw a line between R-like sounds and L-like sounds, the future Japanese speakers draw none and throw all those sounds together into one box. So once you've missed your opportunity to be exposed to English before your first birthday, I'm sorry, the odds are stacked against you. But you can still train yourself. (Jensen 2011; Kuhl et al. 2006. Apparently, the age at which a child stops acquiring phonemes is still much debated, but it's most likely pre-puberty. But I'm no expert on this so somebody may be able to enlighten us.)
Identifying the types of /r/ and /l/
Any phoneme in any language is realized slightly (or sometimes plainly) different depending on what comes before and what comes after. In addition, some phonemes are restricted from appearing in certain circumstances. So it is crucial to be able to identify different realizations (allophones) of the same phoneme, and where they occur, as a step to distinguishing /r/ and /l/.
The environments in which /r/ or /l/ may be encountered can be divided into several groups:
- Initial, as in read/lead
- Intervocalic, as in arrive/alive
- Preconsonantal, as in fort/fault
- Final, as in core/coal, war/wall, litter/little, labor/label
- Postconsonantal, as in fright/flight
The prototypical realizations of /r/ and /l/ surface when they are initial (read/lead) or intervocalic (arrive/alive). They are most typically pronounced as [ɹ] and [l], respectively, as described above. In these positions, [ɹ] is often pronounced with the lips sticking out, like you do when pronouncing お [o]. This makes [ɹ] sound like the sound of English W or Japanese わ行 (which it might do to you in other positions as well, but here even more so), so if you hear a W-like liquid in these positions, you know it's /r/. (In American English, /l/ in these positions often sounds a bit more like [ɫ]—see below.)
You probably know that sort is pronounced the same as sought in the EFL-oriented variety of British English, while they are always distinguished in American. In positions preconsonantal (fort/fault) or final (war/wall, litter/little), AmE /r/ sounds pretty much the same as [ɹ] in initial/intervocalic positions, except it might sound a bit less like W. /l/ in these positions (in both BrE/AmE) is pronounced as a "dark L", [ɫ], with the back of the tongue raised, sometimes with the tip of the tongue not even touching the roof of the mouth, sounding something like う [ɯ] or お [o]. Have you ever wondered why little sometimes sounds like リトゥ ritou or リロゥ rirou? This is why.
I don't think many Japanese-speaking learners of English have difficulty distinguishing war and wall, fort and fault, or litter and little, even in American English, so these instances of /r/ and /l/ are probably not much of a concern to you. Strictly speaking, /r/ in litter/labor and the second /l/ in little/label are not even preceded by a vowel, but I personally haven't seen this being much of hindrance to learners of English. Also, when a word with final /r/ or /l/ is followed by a word beginning with a vowel, the said /r/ or /l/ is treated as intervocalic rather than final (in both BrE/AmE; but you probably knew this too).
Probably the environments in which Japanese learners find it most difficult to distinguish /r/ and /l/ are postconsonantal positions. /r/ and /l/ do not share the same set of consonants after which they can occur.
Consonants after which either of them can occur are:
- /p/, as in pray/play
- /b/, as in brew/blue
- /k/, as in crew/clue
- /ɡ/, as in green/glean
- /f/, as in free/flee
- /sp/, as in sprint/splint
Consonants after which only one of them can occur are:
- Only /r/
- /t/, as in try
- /d/, as in dry
- /θ/, as in throw
- /ʃ/, as in shrink
- /st/, as in strong
- /sk/, as in scroll
- Only /l/
So keeping the latter group in mind will enable you to immediately know whether it's /r/ or /l/ upon hearing those combinations, although you may have already built this intuition so this is not news to you. (There are nonetheless a handful of exceptions to almost every one of them, such as schlock and Sri Lanka.) Other than that, I too have found distinguishing /r/ and /l/ in these environments quite difficult. But there are clues.
One thing we can say about both /r/ and /l/ in postconsonantal positions is that, when preceded by a voiceless consonant /p, t, k, f, θ, s, ʃ/ (provided the said voiceless consonant is not also preceded by an initial /s/), /r/ or /l/ becomes voiceless. That is, the force of the breath caused by the initial voiceless consonant permeates at least the beginning, if not the whole, of the following consonant, /r/ or /l/. This tendency is particularly true for /p, t, k/.
This devoicing process may be part of a clue. After /p, k/, /r/ is realized with much friction, thus producing a voiceless fricative [ɹ̝̊]. One hears a brief friction in /l/ after /p, k/ too, but since /r/ doesn't involve the tongue touching the roof of the mouth, one hears much more prominent friction in pray or crew than in play or clue. I assume this is also true, though to a lesser extent, in free/flee.
In other combinations such as brew/blue, green/glean, sprint/splint, I'm afraid I'm not aware of potential auditory cues other than the quality of each sound per se and the fact that /l/ involves the tongue touching the roof so one might hear the tongue striking the roof at the beginning.
Of course, /tl, dl, θl, ʃl, stl, skl, sr/ still occur beyond syllable boundaries. This would probably not create much confusion, but one thing that may still help to remember is that /tr/ and /dr/, as in true and dream, often sound like [tʃɹ] and [dʒɹ] (as if they're chrue and jream). This happens even within words like petrol and bedroom. /tl/ and /dl/, as in butler and deadly, on the other hand, are often realized as [ʔl] and [dˡl], respectively, both of which probably sound just like a long-held [l] (like バッラー barrā, デッリー derrī).
How to teach (yourself or others)
Lively et al. (1994) found exposure to minimal pairs, like those I mentioned above (read/lead, arrive/alive, fright/flight...), to be effective. To quote from Wikipedia:
Lively et al. (1994) found that monolingual Japanese speakers in Japan could increase their ability to distinguish between /l/ and /r/ after a 3-week training period, which involved hearing minimal pairs (such as 'rock' and 'lock') produced by five speakers, and being asked to identify which word was which. Feedback was provided during training, and participants had to listen to the minimal pairs until the correct answer was given. Participants performed significantly better immediately after the 3-week training, and retained some improvements when retested after 3 months and after 6 months (although there was a decrease in recognition ability at the 6-month test). Reaction time decreased during the training period as the accuracy went up. Participants could "generalize" their learning somewhat: when tested they could distinguish between new /l/ and /r/ minimal pairs, but performed better when the pairs were said by one of the five speakers they had heard before rather than by a new speaker.
According to Mark Liberman (2008), it is crucial to get multiple speakers as samples to listen to for training:
If you test repeatedly on a single example, subjects won't be able to generalize to other examples. If you use just one speaker, then subjects won't be able to generalize to the productions of others. But experience with a few different repetitions of a few dozen example types by each of a half a dozen or so varied speakers seems to be enough to allow generalization to new examples and new speakers.
This method is known as High Variability Phonetic Training (HVPT). Here is a followup from 2019, with a link to a meta-analysis demonstrating the effectiveness of the method by Ron Thomson.