>betraying the slightest suggestion of a smile.  

That means simply that the captain was smiling very slightly, probably without intending to. It shows that he thinks Rostov is naive to believe that these are his quarters.  

[Merriam-Webster "betray"][1]  
 4a : to reveal unintentionally  
 *betray one's true feelings*  

The OP has a quotation from "A Gentleman in Moscow" by Amor Towles.

  [1]: https://www.merriam-webster.com/dictionary/betray