Which can be a determiner (noun follows) or relative pronoun (new clause with subject/verb/complement follows).
In your case it's best if you use it as a determiner, like articles.
I don't know which book you want
is how you want to say it.
I don't know the book which you want
This is technically correct but if someone is not listening closely they may think you are talking about not knowing a book, in the same sense as not knowing a person or subject.
You would say this if you never had heard of the book that the other person is talking about at all.
I don't which book you want.
You broke the sentence here by not finishing your current clause before starting a new one with which. "I don't" always needs a verb right after it.
(You can only say "I don't" without a verb after it in response to a question and nothing else will follow, e.g. "Do you work on cars? I don't.")