What's the correct construction? Say, you're inside the house and are peeking to see what's in the balcony?
Faking courage, I peeked over/out/out of the balcony.
The three examples in your title all mean that you are on the balcony and looking somewhere else from there. I think that what you mean to say is
Faking courage, I peeked out (of the house) onto the balcony
Faking courage, I peeked into the balcony
if it's more of an enclosed space and almost like a room.