If I am asking a friend about an event they invited me to in a few days is it wrong to say 'What time did you say it was?'
American English speaker here, and this is exactly the way I'd ask the question; both tenses shift to the past. However:
What time did you say the party is?
Is also correct, with no difference in meaning. I might be more inclined to the use the present tense if it's already the day of the party.
There's a distinction here between spoken and written English. In spoken English,
He said the party starts at 9 o'clock.
sounds absolutely fine. In written English it would look wrong or like dialect; it should read:
He said the party started at 9 o'clock.