The usual idiomatic expression is 'to lower the tone of' something. This means that you make something (e.g. an event, a conversation, a location, etc.) less polite, or less dignified, or less elegant, or less respectable than it should be.
Collins English Dictionary defines 'lower the tone' as:
If you say that something lowers the tone of a place or event, you mean that it is not appropriate and makes the place or event seem less respectable.
Your excerpt is using a slight variation on the idiomatic phrase, substituting 'pull down' for 'lower', but the meaning is the same. So, your interpretation, that 'pull down the tone' means to 'spoil' is correct.