As per the grammar rules, if "I" is the subject of the sentence then the other person must the object. And generally whom is used in the objective form but in this case the prior format is more common than the latter. So which one is correct?
Much depends on location and audience. Some English speakers expect the proper use of "whom", and wince when ending a sentence with a preposition. Others could not care less.
While I know how to use "whom", perhaps because I'm American, I rarely use it. In most situations it sounds too formal, or even (to some extent) pretentious.
Then again, "Who are you spending it with?" already sounds somewhat stilted and formal, so in any situation where I might say something like that, I would probably feel obligated to use "whom" and reorganize the sentence appropriately. Without more context, I couldn't say what the informal version would be, but possibly something like:
Who are you going with?
Who is going with you?
Who else will be there?
and so on.
Again, it's a personal choice. Other English speakers might feel your example is perfectly ordinary conversation.
"With whom are you spending it?" is the "most correct". But in real life, when speaking, one would say "Who are you spending it with?". The word "whom" is very unusual in spoken text nowadays; most people use "who" for all cases, not just the nominative.
From the OED:
Although there are some speakers who still use who and whom according to the rules of formal grammar as stated here, there are many more who rarely use whom at all; its use has retreated steadily and is now largely restricted to formal contexts. The normal practice in modern English is to use who instead of whom (and, where applicable, to put the preposition at the end of the sentence): who do you wish to speak to?; who do you think we should support?