Both versions given are grammatical. The prepositional phrase "by him" is acting as an adverb, and both "[verb] [adverb] [subordinate clause]" and "[verb] [subordinate clause] [adverb]" are valid. Yours is more idiomatic, partly because your teacher's version has a slight ambiguity.
The subordinate clause "to solve the problem" is of the form "[subordinating conjunction] [verb] [object]", and so "What was done to solve this problem by him?" could be interpreted with the adverb-phrase "by him" attached to the verb "to solve". This has a different meaning, in that "he" is the means of solution, but the agent of the solution (the person doing the things) is left unspecified. This disagrees with both the active form given and your suggested passive, wherein "he" is unambiguously the agent, and the means is not specified. Native speakers would be very unlikely to interpret the sentence that way, but the grammar permits it.