Joyce's use of English is highly varied and sometimes quite strange, but as the comment by Kate Bunting indicates this seems to be a simple expletive, in which an ordinary word is sued as an expression of annoyance, rather than a profanity or blasphemy that many people might use in such a situation.
I remember in several of the stories of Saki the word "Rats" is used in a similar way, In particular in "Tobermory" a character thinks but does not say "Rats" and the narrator refers to them as "Those rodents of disbelief".
As the comment by
Michael Harvey
points out, the expression "Oh Rocks" is frequently used by the charactere Molly Bloom. In "James Joyce's concept of the underthought: a reflection on some similarities with the work of Wittgenstein" by Mike Harding the author writes:
I still regard Joyce as the greatest creative novelist, but after having logged on to various websites to see the current state of Joycean play, I logged off fairly swiftly with Molly Bloom's favourite expression in mind: Oh, Rocks!
I would mention that both older usages, intentional wordplay, and the assumptions that the reader knows classical, biblical, and other literature, make Joyce a sometimes confusing author for any modern reader, and particularly for a learner.