As a German speaker, I'm surprised to hear by myself/yourself/himself... or on my/your/his... own where I would have expected the word alone or myself/yourself/himself... (without by). Meanwhile, I even started to doubt if I ever should use the word alone in English. But on the other side, the English word alone exists, so it must be idiomatic in some cases.
So, my question is:
When to use the word alone?
When to use by myself/yourself...?
When to use on my/your... own?