Neologism is defined as: a new word, expression, or usage. In mathematics, there are new symbols created every year, and new usages for those symbols. Is there a word for a new symbol or usage, like neologism exists for word? If there isn't any word for it, how would you refer to the idea?
I am not aware of any commonly used word for a newly coined symbol - usual practice is simply to provide a definition
p = J ≀ R where ≀ denotes the wreath product
For a more extensive novel usage, something like:
Throughout this paper we introduce the notation J ≀ R to mean the wreath product of groups J and R.
If however, you are discussing more generally the use of symbols in mathematics (for example, in a paper on typesetting problems) you can refer to "newly introduced symbols", "newly coined symbols", "newly adopted symbols" etc. This will become cumbersome if it is used much. It would be acceptable in those circumstances to coin or adopt an appropriate word ("neoglyph" would probably work) and make it clear what meaning you attach to it, in the context of your paper.