The normal use of "out of keeping" does use the preposition "with," as in "loud talking is completely out of keeping with proper library etiquette." With this idiomatic phrase, two things are being compared: loud talking and proper library etiquette, and they are found to be incompatible.
But this sentence doesn't use "out of keeping" to compare anything; it just uses it to mean something like "wrong" or "inappropriate." So it says "in this library" to describe the location where loud talk is "out of keeping." Other examples where "in" could be appropriate:
Yelling is out of keeping in an exam room
Insulting your classmates is out of keeping in English class
That said, this sentence is close to unreadable to this native speaker, and I wouldn't recommend trying to learn too much from it. A few ways it might be rephrased:
Loud talk is out of keeping with library etiquette
Let's keep loud talk out of this library
[Please] no loud talking in this library