Can I say "I’m afraid yes" to mean "I'm afraid so"?
In these conversations, you are B. Read the information in brackets and then answer with I think so, I hope not etc.
(You’re at a party. You have to leave early.) A: Do you have to leave already?
B: I’m afraid yes.
The correct answer in the book B: I’m afraid so.
Source English Grammar in Use Intermediate Unit 51.4.6