When you are invited to the party from your boss and would like to join, is it appropriate to say "I'm in"? Does it sound too casual in a business setting?
I'd like to know more formal ways of saying.
Only you can answer this properly. That's because, like with any conversation, it depends entirely on your relationship with the other person.
As it happens, just a few weeks ago I literally replied "I'm in!" to a party invitation from our HR manager, but it's a small company at which I've been working for thirteen years. My colleagues are also my friends.
I wouldn't recommend responding in that manner to a senior manager who you don't know socially, because it is medium informal.
The other extreme is:
Dear Mr Biggins,
Thank you for your kind invitation. I am delighted that I shall be able to attend this event. Please let me know what the dress code is: black tie, or will simple business-wear be acceptable?
It depends on the type of party. If it is a formal dinner party, no. But if it is an informal get-together for a drink after work, you can say "I'm in" or "Count me in!" which would be an expression that you are happy to come along.
If it is a formal business setting - such as a formal dinner party that is too casual. Especially as it is your boss, it is always best to err on the side of caution. A phrase such as "Thank you that would be great" would be better.
For a Friday night drink or informal setting, "count me in" would be fine.
It depends on how you want to portray yourself.
Because the conversational situation you are describing is
1) informal (someone you know well)
2) you are being invited to a party
3) you are talking to a co-worker, even though it's your boss
4) assuming you are on good terms with your boss
is appropriate, of course adding "thank you" is courteous
Thanks for the invite, I'm in!
would convey a certain excitement of looking forward to going to the party. If you are too formal you may come across as being aloof or distant.