This is actually a very good question.
The two examples you gave are not negative (i.e., they do not contain negated verbs), and nor is generally limited to following negative sentence, that is true.
However, the ‘negative’ part of that is sometimes more semantic than syntactic/grammatical. If the meaning of the preceding sentence has an element of being negative in nature about it, then you can use nor, as long as it makes sense semantically to do so.
In both your example sentences, there is an element of negativity: refusing is a ‘negative action’ semantically (the negative of assenting), and missing a train can just about be considered one as well.
Your second example to me sounds perfectly fine, and it is something that many native speakers would say without a second thought. Your first example sounds a little bit odd, but I suspect that's due to the lack of parallelism between the two sentences in it: what does leaving have to do with listening to you? If you had chosen a more parallel example, there would be no problem; for example:
He refused to talk to me; nor would he listen to what I had to say.