I can make sense of verbs like turn (the TV) on and take (a tooth) out, as the on and out parts mean working and outside respectively. However, the contribution of the preposition/adverb to the overall meaning of the phrasal verb is not always obvious. For example, he came off as unconcerned. What meaning and role does off play towards the meaning appear ?
As native English speakers, it's not always clear why a phrase is the way it is, but the meaning is clear.
As far as I understand it, the off refers in a sense to an aura or appearance of a person. So came off is akin to "his demeanor" or "his aura emanated (from him) as unconcerned".
Come off therefore has a sense of moving from one place to another. One person's appearance or emotional state makes it's way to another person's senses.
As you've correctly pointed out, phrasal verbs often have figurative or idiomatic meanings, and while sometimes the meaning is somewhat intuitive (or at least, one can find some sort of logic behind it), other times it is not, and to add to the confusion even further, some phrasal verbs also have a literal meaning, as well as a figurative one.
Consider the example you have given - come off
The paint came off the wall.
The phrase has a quite literal sense in this example - the paint, quite literally, is not on the wall anymore - it fell off the wall. And the interpretation of this is quite logical for most people.
The rider came off his horse.
The usage here becomes more figurative, meaning to fall off something that you are riding. However, it's still reasonably intuitive, in that most people would be able to figure out that horse and rider had become separated somehow.
The man came off as a little desperate on the date.
Now we're stepping into purely idiomatic territory - 'come off' now means to act in a way such that people form a particular opinion of them. There's no logical connection between the phrase and its meaning as neither come nor off seem to have any association with how one might act.
- The party didn’t quite come off as we had hoped.
Again, there's no logical connection between the phrase and definition - this time it means to succeed. Again, neither come nor off seem to have any direct link to success.
Unfortunately for English learners, there are quite a few phrasal verbs like this, and there really isn't any pattern to them in terms of rationalising them, other than to memorise them.