There's usually no significant allusion to "unavoidable misfortune" with the cited construction. Compare, for example,...
1: I'm sorry you had to hear that
...which nearly always means...
I'm sorry you happened to hear that
(you weren't supposed to hear it, but unfortunately you did).
The only reason you don't normally hear...
2: I'm sorry this happened to happen to you
...is because it's an ugly repetition. But a long time ago we might have said...
3: I'm sorry this chanced to befall you
...with exactly that implication.
TL;DR: Including had in the cited context usually makes no difference to the meaning. But it's idiomatically well-established for the context, and at I think at least some people (besides myself) would agree that if it is included, there's at least a slight implication that whatever happened, was heard or seen, etc. was an unlikely / unexpected / surprising / unfortunate accident.