If it's too early to do something, that means the proposed activity normally takes/ should take place later within the contextually relevant time-frame. That's later in the day for OP's exact example, but I might say it's too early [in the year] to be planting flowers in your garden right now, for example.
On the other hand, too soon normally means not enough time has elapsed since some contextually relevant earlier event. No matter what time of day you make your move, it might be too soon to start chatting up that woman whose boyfriend died only last week.
There's some degree of "interchangeability" between the two usages. In particular, it's not at all unusual to use too early for the second meaning above. But those are the most common implications (so most native speakers would use too early in OP's context).