You have the idea exactly; tomorrow is considered to be in reference to the present. More formal versions of the same idea are:
You said that the finals were going to start tomorrow.
You said that the finals would start tomorrow.
What "you" actually said was one of these:
The finals start tomorrow.
The finals are going to start tomorrow.
The finals will start tomorrow.
Of these, the first one is informal; since we all know that we are talking about the future, the future tense is implied rather than stated. Taking the present tense verb and placing it in simple past has the same level of informality.
Because of this ambiguity of tense, you will often also see
You said the finals start (are starting, are going to start) tomorrow.
I would call this equally correct. Thinking about it, these have a slightly different emphasis to me:
He told me the finals started tomorrow, but it looks like they're going to be rained out.
He told me the finals start tomorrow, so you'll need to get your tickets today.
The first one suggests that the finals are not actually starting tomorrow on schedule. The second one suggests that they are.