could is about possibility- it is the past of can, and it's about being able to do something. can't and couldn't means that something is impossible.
must is about probability- you use it when you confidently infer something from information that you have.
- Martha couldn't have taken your notebook, she wasn't even in class.
This means that it is impossible that Martha could take your notebook. She was not able to, because she was not in the class.
In this sentence, you would use couldn't because it really was impossible that Martha took the book. It's not a matter of probablility, it's a fact. I will therefore use a different example:
2a. Martha missed the class: she must'nt have heard that the time had changed.
2b. Martha missed the class: she must have not heard that the time had changed.
We infer that Martha probably did not know that the class time had changed. The information that we have is that we know that she missed the class. Most people would choose the second form, with not after have.
Neither of the Policeman examples work, so I will move on to the "right answer" examples.
- That mustn't be right answer
Imagine that you are doing a crossword puzzle. You have filled in an answer for 1 across, then you read the clue for 1 down and you are sure that you know the right answer for it, but the first letter isn't the same as the first letter of the answer for 1 across. So you say example 3, which indicates that you infer that the answer to 1 across is probably wrong. The information you use is that you are sure that the answer to 1 down is right, and that doesn't fit with this answer for 1 across.
- That can't be the right answer
Imagine that you are solving a math puzzle and the question says that the answer must be an even number. You do the sums and get an answer of 13. You know that the answer must be an even number, so is is impossible for 13 to be the right answer.