I wonder the sentence "Tomorrow might be difficult." is natural.
For meaning like this context :
A : how about having dinner with me tomorrow?
B : Tomorrow might be difficult. I have to do work.
Thanks for your answer in advance!!
Yes, this is an idiomatic way of saying
It might be difficult for me to (have dinner) tomorrow.
Similarly you can say
Tomorrow is good.
Tomorrow can function both as a noun and as an adverb.