During graduation ceremonies at such universities (including those in the USA), students, faculty, and other dignitaries wear academic gowns, such as you see below.
Wikipedia offers this explanation:
The academic dress found in most universities in the Commonwealth of Nations and the United States is derived from that of the universities of Oxford and Cambridge, which was a development of academic and clerical dress common throughout the medieval universities of Europe. This overgarment had the practical purpose of keeping a scholar warm while they were sitting, immobile, studying.
The gown refers to the students and professors who wear the gowns. Traditionally, they belong to the wealthy, educated class. The town refers to the working-class people who live in the town where the university is located. They might be employed as housekeepers, cooks, gardeners, etc. at the university itself, or they might be farmers, factory workers, shop assistants, and so on.
The point of the phrase town and gown is to draw attention to a distinction between these classes. Today, the perception that they are different can exist even where the college or university attracts working-class students. My personal experience might be unusual, but I have never heard the phrase used in a positive way. I have usually heard it in contexts where the university president says things like this:
We have a serious town-and-gown problem. The students treat the locals like dirt. The mayor is so upset he won't let us hold our annual parade. I need you professors to talk some sense into these kids.