The clause in question is buried fairly deeply in its sentence. It may be easier to examine the structure in an independent clause:
I do not want you dating my wife.
This clause has a subject (I), a verb (do want) modified by an adverb (not), a direct object (him) and an object complement (dating my wife).
Your question also provides another grammatically correct option:
I do not want you to date my wife.
These two example independent clauses have the same structure: subject / verb / direct object / object complement. The difference lies in the grammatical form of the two complements. The phrase "dating my wife" is a participial phrase.* The phrase "to date my wife" is an infinitive phrase.
Participles and infinitives are quite similar. They are both non-finite forms. They both can act as modifiers. In these examples, they both modify the direct object through the action of the governing verb.
As non-finite forms, participles and infinitives do not have a tense. That is to say, they are not limited to (or placed in) the past, the present, or the future. Despite having no tense, they do have aspect. Infinitives have the indefinite aspect, while the so-called "present" participles have the continuous aspect.
Because this participle has the continuous aspect, "dating my wife" seems more concrete or more immediate than "to date my wife". In the context of the episode, this makes sense. That Penny and Zach are married is a fact, and that Penny and Leonard are dating is also a fact. These are the concrete and immediate circumstances that Zach references when addressing Leonard. The infinitive would be more appropriate if either the dating or the wife were hypothetical.
Zach's objection isn't limited to Leonard. Rather, there is some kind of guy that Zach would not find objectionable, but Zach is starting to doubt that Leonard is such a person.
I want one kind of guy to date my wife. I'm starting to think [that] you're not that kind of guy.
In the original line, all of these clauses are combined into one sentence:
I'm starting to think [that] you're not the kind of guy [that] I want dating my wife.
Each "that" in brackets above is optional. The first doesn't play any role within the clause that it introduces. The second is a relative pronoun, and it serves as the direct object of its clause.
This is the other reason that the phrasing "I want dating my wife" may seem strange at first glance. On its own, it doesn't work as a clause or even as a coherent phrase. The direct object is missing. Its existence is implied by the way the clause follows the noun phrase which it modifies.
"Dating my wife" is a participial phrase, consisting of the so-called present participle "dating" and its direct object "my wife". The whole participial phrase is the object complement in the clause "I want [that] dating my wife". This relative clause is used as a contact clause to modify the noun phrase "the kind of guy", leaving the optional direct object [that] unspoken. In turn, "the kind of guy [that] I want dating my wife" is the subject complement of its clause, which in turn is the direct object of the outermost clause.
Here are the answers to your two-part question:
"You are not the kind of guy I want to date my wife" is a grammatically correct sentence, and it means nearly the same thing as "You are not the kind of guy I want dating my wife". It is also grammatically correct to say "You are not the kind of guy I want my wife to date", but this arrangement makes "my wife" the direct object of "want" rather than "to date". Again, the meaning is practically the same, since we naturally assume that dating implies a reciprocal relationship.
The verb "want" takes a direct object. I can't recommend using the preposition "for". Prepositional phrases do not make good direct objects. We could consider "want for" to be a phrasal verb construction, but that would imply a different meaning which is closer to "lack" or "need" than it is to "desire" or "intend".
"You are not the kind of guy I want dating my wife" uses a contact relative clause with an implied direct object and an explicit participial phrase modifying that implied object as an object complement, as licensed by the verb "want".
* Without context, the phrase "dating my wife" could be either a participial phrase or a gerund phrase. Gerunds can act like nouns. Participles can act like modifiers. The only way to distinguish between "present" participles and gerunds is to see what role the phrase plays in its clause.