The second sentence has a relative clause without a relative pronoun. That's allowed, but for clarity the second sentence is equivalent to
Mary and I are going to meet John in the school that Mike used to work in.
If you omit "in" you get "the school that Mike used to work". This implies that "Mike worked the school.", which is grammatically correct nonsense. So you would need "in" (or "at") for the second sentence to have a meaning similar to the first.
The first sentence is much clearer. It has a clear relative pronoun to indicate location and is more elegant. The second sentence, without a zero pronoun, would make me stop to work out how to parse it. It's a poor piece of writing.