The highlighted phrase in the paragraph
He had considered walking north on Fifth Avenue from Schrafft's to a
point from which it would take him until ten o'clock to reach the
house. At that hour people were less likely to be coming in or going
out. But the procedure would have made an awkward loop in the straight
thread of his casualness and he had abandoned it. It was impossible to
figure when people would be entering or leaving the house, anyway.
There was a great risk at any hour.
He wanted his route of walk to look casual, not planned in advance. Why? So that he could explain his appearance near his victim's house easily: "I was just strolling after my meal, and struck upon her house by accident."
One casual way to walk is to just walk straight ahead, making no turns. Say, if one is not looking around but is thinking about something ("absent-minded").
Walking to a certain point on Fifth Avenue and then turning to go to her house could've been interpreted as a premeditated plan of action; at least so he thought.
There might be a note of irony lurking in the combination "straight thread of his casualness": straight thread has a tinge of deliberateness, whereas casualness is associated with something erratic, haphazard. The author might hint at the hero's predisposition to order even in off-work situations.