"We are going to our houses?" (the house is not the same for both speakers) I feel like there is something wrong with it. As if it's incomplete, or missing something..
It's a legitimate sentence, but it's not very idiomatic. There is an English expression (and a song)
A house is just a physical structure, while a home is lived-in (often by a family) and full of memories.
While it is fine to talk about going back to your house, it is more common when talking about your own house to refer to it as your home. As Lambie suggests:
We are (all/both) going home.
Note that "home", as a physical structure, is countable:
Don't you boys have homes to go to?
but as a concept, it is uncountable.
You boys should all go home (each to your own, individual homes)
Home is where the heart is.