According to English Thesaurus, the meanings of set off, can be, among the many meanings, compensation or departure. In this case set off means depart, leave... Does it sound crazy? Together
Together, means, among other meanings: synchronism (according to English Thesaurus), when adverb(the main meaning is accompaniment, junction, union) but it can means in some cases: synchronism (according to English Thesaurus), when adverb. In this phrase together means in context at the same time. So the meaning of this phrase can be understood as:
The two then set off (leave) together (at the same time), alone.
In this definition, together can mean at the same time in some cases