I couldn't understand the words they have used.
Look at the above statement carefully, it has made of two different times.
'I couldn't understand' is one and another is 'the words they have used'
'I could not understand' talks the simple past. However, 'the words they have used' is where you need to look at. Have + past participle is used when something happened in past and is still going on or simply has an effect in your present (current time)
Look at this one: I have lost my car keys. This simply means that you haven't found them yet or you still be standing out of your car and looking for it. "This is your present (current time) situation.
'I couldn't understand the words they have used' simply means you still don't know what they said.
If you say, 'I couldn't understand the words they had used' means you don't need to know now, it was a point in time where you needed to know but 'now' you don't.
I would rather suggest you to look at the time lines of the English grammar tenses.