“Denoting A the set” or “denoting by A the set”

In mathematical language one often uses the phrase

Denote by A the set with whatever properties.

and similar uses of the word "denote". (See for example Is it correct to use an expression "we denote something by x"?.) Let us accept this usage as correct.

Can the word "by" be omitted? The sentence would then read

Denote A the set with whatever properties.

"To denote" only licenses a direct object.  It doesn't license an indirect object.  It doesn't license an object complement.

In canonical form, "A denotes the set with the given properties".

When we transform this to an imperative, we have a problem.  There is no third-person imperative in English.  The subject of an English imperative is always the second person, usually the implicit second person.

We can transform "to denote" to the imperative mode.  However, we no longer have a subject that can act as the semantic agent of this verb.  In order to include that agent -- the symbol that denotes the set -- we have a couple of options.

We can use a prepositional phrase.  The prepositions "by" and "with" are suitable for that purpose.  That brings us back to your original example.

We can set some other verb in the imperative mode, and let the infinitive form of "to denote" modify its agent.  In that case, the sentence takes this form:

Let A denote the set with the given properties.

We could, of course, avoid the entire issue by choosing a different verb that does license an object complement.

Declare A the set with the given properties.

or

Define A as the set with the given properties.

The phrase denote «object» by «symbol» means the same thing as We will refer to «object» using the name «symbol». In English, it is permitted to “reverse” this to denote by «symbol» «object» without changing the meaning. I have also heard the usage denote «object» as «symbol», with identical meaning, but this is unusual, and I believe that it is technically incorrect as well. In any case, to omit by suggests that «symbol» is actually «object», and would be incorrect for this context.

ref: Merriam-Webster’s Learner’s Dictionary, denote
ref: Merriam-Webster dictionary on-line, denote
ref: Oxford dictionary on-line, denote
ref: Cambridge English Dictionary on-line, denote