If one of the adjectives is a superlative, by definition this implies there are others of the same general type. So the youngest small man implies there are other small men who aren't so young, and the smallest young man implies there are other young men who aren't so small. In both cases, the highlighted word pair is "elevated" to the status of a known combination (because we know there are other contextually relevant instances of the same general type). Hence the non-superlative part of the noun phrase becomes "unbreakable", and must appear last. The last two words together become a kind of "compound head noun".
That's to say - the small youngest man and the young smallest man are incredibly unlikely sequences, because they would require that the audience be familiar with the youngest man or the smallest man as meaningful references. Which might occur in a contexts like the young top dog, because "top dog" is a known concept. But there won't be many contexts like that.
For a more obviously "superlative" context...
He was a lucky fastest loser in the initial qualifying rounds, but he went on to win the championship
...where fastest loser is an established collocation (as defined in the link above), and fastest lucky loser would be nonsense.