In an old style phone the main part of the phone was mounted on the wall and there was a separate part, connected to the phone by a cable, which hung on a curved Y-shaped lever. To make a call or answer a call you picked up the separate part and a spring lifted the lever and changed the connections inside the phone. To end the call you hung this part back up so its weight would return the phone to the waiting state.
Hence "hang up" the phone and "pick up" (answer) the phone.
There was no equivalent to switching the phone off unless you unplugged it, as its electricity supply came along the phone line.
To hang up a phone therefore means to end a call, whether you have had a conversation or even if the other phone has not yet been answered.
To switch off the phone is to turn it completely off. This would end a conversation, but it is not usually used for this purpose, partly because many mobile phones require you to hold the power button for several seconds and then confirm that you wish to turn the phone off.