"To denote" only licenses a direct object. It doesn't license an indirect object. It doesn't license an object complement.
In canonical form, "A denotes the set with the given properties".
When we transform this to an imperative, we have a problem. There is no third-person imperative in English. The subject of an English imperative is always the second person, usually the implicit second person.
We can transform "to denote" to the imperative mode. However, we no longer have a subject that can act as the semantic agent of this verb. In order to include that agent -- the symbol that denotes the set -- we have a couple of options.
We can use a prepositional phrase. The prepositions "by" and "with" are suitable for that purpose. That brings us back to your original example.
We can set some other verb in the imperative mode, and let the infinitive form of "to denote" modify its agent. In that case, the sentence takes this form:
Let A denote the set with the given properties.
We could, of course, avoid the entire issue by choosing a different verb that does license an object complement.
Declare A the set with the given properties.
or
Define A as the set with the given properties.