One possible explanation is that your intonation was off. If you put emphasis on the "boy" as a separate word, then it could be assumed your addressing the person as "boy". The two words should be pronounced as if they were one word, "ohboy".
Another is that the expression is slightly misused. "Oh boy" is generally an internal expression of surprise, and not usually directed at anyone else. If someone else can hear you, and might think you are talking to them, then "oh no", or "excuse me" is better.
Lastly the person might simply have misunderstood you, or deliberately misconstrued your simple expression to imply you were addressing him as "boy". Ordinarily if I do this, it's with humor and grace, in order to defuse a potentially embarrassing situation, but other people might prefer to respond with indignation.