Why is the letter Y in 'by' pronounced 'bai' but in 'Tyrion' it's 'Tirion' and not 'Tairon'?
Both pronunciations of Y are correct and often found in English, because English is influenced by many different languages. Many of its words have decidedly different origins and pronunciations, but because English only has so many characters in its alphabet, different sounds inevitably get represented by the same letters sometimes.
Examples of /aɪ/ ("long I") pronunciations:
Examples of /iː/ ("long E") pronunciations:
In fact, Y can be pronounced other ways as well. For example, synonym pronounces Y as /ɪ/ ("short I", as in kit or lid).
As for by and Tyrion specifically, the pronunciations differ because the words have different etymologies. By comes from Old English, which is a West Germanic language 1, 2, while Tyrion (I assume, I don't know the thought process behind the writer of Game of Thrones) comes from Tyr which is North Germanic/Old Norse.
The two words are also structurally different. Tyrion has multiple syllables while by has only one; the Y in by is at the end, whereas in Tyrion it's not; etc. All of these things can influence pronunciation.