Past perfect is mainly used to convey a sequence of events. The past perfect implies that one action was finished/was going on before another action. In all cases, simple past may suffice, except in cases where some emphasis of sequence is important.
In your case, the following composition is just fine:
"he asked me why I didn't call him all day. I said I was busy all day. I visited visited a local temple first thing in the morning. Then I went to yoga class with Margaret. After class we both went to a restaurant for a breakfast. I met met my boss there"
However, the second sentence in the following composition emphasizes/clearly establishes a sequence of past events that happened before 'asking'. The speaker is trying to convey that that he was busy every moment up to the moment of 'asking'.
"he asked me why I didn't call him all day. I said I had been busy all day. I visited a local temple first thing in the morning. Then I went to yoga class with Margaret. After class we both went to a restaurant for a breakfast. I met met my boss there".
You do not need past perfect in all events because sequence of events is already clear from sentence to sentence. Please note that past perfect is usually avoided when specific time in the past is mentioned. So, I visited a local....is preferable to I had visited....in the morning.