"I didn't have the phone to myself yesterday"
Is that a correct sentence and a way of telling someone that your phone was not with you yesterday and that is why you couldn't reply
To have something "to yourself" means to have it for your exclusive use.
"I did not have the beach to myself - there were several other people on it".
The example sentence you give means that you were sharing your phone with other people.
It is normal and correct to say
I did not have my phone with me.
The sentence is not necessarily wrong. In the days when people did not carry telephones around with them all the time, it was quite common for a single phone to be shared by numerous people. In those circumstances, it would be a very reasonable explanation of not being able to receive a call (possibly of a very private nature) that one 'did not have the phone to [oneself]'.
If the sentence referred to a time in the 1950s, say, it would be fine (if the intended meaning was as I have indicated).