There is a significant difference in meaning if you are talking about the pen and the book as objects that you are just locating in space... vs. getting a pen ready and poised to write in the book.
Let us say the pen is just an object that might serve the purpose equally well whether it is capped or uncapped, and its nature as a writing tool is irrelevant. Then simply upon/on would likely mean the pen is resting on the book and touching, due to gravity. above would probably suggest not touching but being in a somehow "upward" direction. (Like hanging from a string in three dimensions, or perhaps placed on a table "above" the book as it is seen from a photo in its defined orientation.).
The construction up the book is bizarre; it suggests some kind of slot into which the pen could be put. You might argue that if a closed book were pierced through the pages along the binding that someone "stuck the pen up the book" but that is awkward. up the (whatever) has "vulgar" associations as a phrasing for almost any (whatever)--at least today.
If it is an instruction about a pen in readiness to be writing in a book, then perhaps the more important thing to point out is the nuance of "book" itself. The word in isolation is typically used to mean something pre-printed that you aren't to mark up...and there are special categories invoked for the kinds of books you are "supposed" to write in (workbook or coloring book).
In that sense the best answer may be "Keep the pens off the books, or you'll hurt their resale value." :-)
If you wanted to tell someone to get their pens ready to write in a book, you'd be clearest saying:
"Open your workbooks to page 3, and touch the tip of your pen to the center of the page."
Because if that is what you are trying to say vs just positioning objects (as for a photo shoot, like @J.R. suggests) then none of the choices you mention convey that meaning well.