I don't think 'layman' is an offensive term by any account. If you go by OALD, it says...
a person who does not have expert knowledge of a particular subject
So, the term 'layperson' separates a common person from an expert. And, being a 'common man' here does not offend you or me at all.
The example down there puts light on it
a book written for professionals and laymen alike
But I do think that replacing it with 'ordinary' may get a few (including me) some note of offensiveness. I'd use common over ordinary.
However it may be worth noting that (at least in British English) 'layperson' seems to be the more accepted term, likely due to maintaining gender neutrality.
