According to English Thesaurus, the meanings of *set off*, can be, among the many meanings, compensation or departure.
In this case *set off* means depart, leave...
Does it sound crazy?
Together, means, among other meanings: synchronism (according to English Thesaurus), when adverb. In this phrase together means in context *at the same time*.
So the meaning of this phrase can be understood as:

> The two then set off (leave) together (at the same time), alone.