I'm a web developer. I work exclusively with Americans, and when working remotely, we all use Slack. I'll have a habit of replying to yes/no questions with "Yes", while sometimes adding a reason why, e.g. "Yes, we do it that way because the business requires us to." Recently, I had my yearly review with my manager, and he told me that while my English is good, I can tend to come across as terse. I asked him for some examples, and he said that the primary example is that I'll reply with "Yes". He implored me to use a more casual "Yeah" when replying to others, as "Yes" can come across as dismissive and can make it seem that their question is stupid or that I don't have patience for their question.
I take everything in my reviews to heart. I know I can improve my english skills further, but I didn't realize that this was a thing I had to be wary of. I asked my technical lead (one level above me but one level below my manager) about it, and he told me that my boss was being overly sensitive and to ignore the feedback. Should I make a conscious effort to reply exclusively with "Yeah", or is "Yes" appropriate to use?
It may also be worth noting that none of my other coworkers have said anything about me being terse. I didn't post this on The Workplace stack exchange because I feel that this is a language barrier issue with me and not a general workplace question, and I want to ensure that I'm being polite when I'm talking to my boss.
Edit: This is a different question than this existing question because the question is unrelated to my issue and only defines the difference between "yes" and "yeah"