Could you paraphrase this sentence in an easy style? : " The Scarlet letter,"he states flatly in a one-sentence paragraph, "had not done its office." Is there anything to do with office?
As you guess, in this case "office" has nothing to do with any physical structure or location. This actually more closely related to the original meaning of the word.
office (n): mid-13c., "a post, an employment to which certain duties are attached," from Anglo-French and Old French ofice "place or function; divine service" (12c. in Old French) or directly from Latin officium "service, kindness, favor; official duty, function, business; ceremonial observance"
You can still find this definition in many modern dictionaries, although this use today feels a little archaic or at least formal. Over time, "office" has come to refer more to the place where one performs official duties, than the duties themselves.
Still, it's not that uncommon:
Although he has been popular, many feel the president lacks the dignity expected of the office.