Let me start with a little observation. In my own experience in various languages, prepositions are pretty arbitrary. In each language, senses get grouped in one preposition in a way that seems natural to native speakers but not to newcomers. So I cannot imagine a very thorough answer to your question. I think that the best we can hope for is to situate each of these expressions among other expressions that seem comparable to native speakers.
There are lots of English expressions that have to do with a mental or emotional state that involve “in,” not “on.” So a child doing something “in tears” seems kindred to someone doing something in a rage, in a confused state of mind, in a fit of pique, in a malevolent spirit, in blissful ignorance, in the belief that it was legal, in hopes of eventual recognition, or while in a good mood. I guess that a judgment “in my opinion” is also something like this. I don’t know why “in” is used in expressions like these, but I cannot think of any expressions like these, involving “on.” Maybe we think of the mental or emotional state as being like a cloud enveloping a person.
I don’t how to analyze the expression about being happy in their marriage. I wouldn’t say that someone is happy in chocolate; the syntax seems to be that we are happy with or about something, not in something. Maybe the word in here suggests that they are happy while they are within their marriage, i.e. while they are married. I’m not happy with that analysis, but I haven’t got a better one.
“He was not thinking well on that occasion” seems comparable to other expressions about when something occurred: on Wednesday, on Obama’s watch, on Thanksgiving, etc. Perhaps some uses of “on your recommendation” are like this, if the sense is “once you have recommended it.”