Regard is tricky. The ‘rules’ below are those observed in formal writing; you may safely ignore them in conversation.
When used as a noun to express your attention to a particular topic, it should be used in the singular:
In regard to the appointment, yes! ... not regards.
With regard to the appointment, yes! ... not regards.
This is often confused, even (or perhaps particularly) by native speakers with the related phrase as regards:
As regards the appointment, yes!
This is not a plural, however, but a 3d person singular verb; the sense is that your discourse now regards (“looks back to”, “contemplates”) the noun or NP which follows. The verb sense is also used in the present participle regarding:
Regarding the appointment, yes!
The plural noun regards is used only in the sense “expression of esteem or affection”. In this sense the plural varies freely with the singular.
Please convey my very sincere regards [or regard] to your mother.
“Give my Regards to Broadway” —song by George M. Cohan
In the sense of “esteem, honor, opinion”, however, only the singular is ordinarily employed.
I hold him in the highest regard.
Respect, which also acts as both a noun and a verb, works exactly the same way in these senses.
With respect to ...
With regard to ...
As respects ...
Convey my respects to ...
Both words have other senses, however, in which their use varies.