Well, neither one is a complete sentence, but you probably realize that. Let's go ahead and make them complete sentences anyway:
I am not sure does it work.
I am not sure if it works.
The only correct sentence is 2.
In 1, "does it work" is a complete question, also known as an interrogative clause. Notice the inversion of "it" and "does," which are the subject and the modal auxiliary verb, respectively. This is a sure sign of a question. In this form, it needs to stand by itself: "Does it work?" Rearranged into a regular declarative statement, it would look like this: "It does work." So "does" is just part of the verb phrase. It doesn't take on any conjunction-like duties.
In other words, not only is there no subordinating conjunction such as "if" to join "does it work" to the main, independent clause, but interrogative clauses can't be subordinated (although they can have subordinate clauses of their own, as in, "Are you sure that it works?").
(The subjunctive mood is not present in either 1 or 2.)