Short Story: 'Residential College' is the most specific term, but 'dorm' is the one most likely to be understood.
"Residential College" is apparently an official term used to describe the sort of institution you're describing in America.
If you had said this to me prior to me reading the answers on this page, I would not have know what you were talking about, and I expect many American speakers would feel the same. "Residential College" sounds like a place where I take classes related to residences. I don't know what that would be, but perhaps it would be related to construction or hospitality.
After hearing your description and reading about institutions that use this term, I would describe them all as 'dorms', which is short for 'dormitories'.
This term is less specific, and does not necessarily imply as much organization and control as what you're describing, but definitely includes what you're describing.
I went to Iowa State University, where I lived in the dorms all four years due to a scholarship that paid for them. Some of the dorms are highly organized. For example, freshman computer science students can choose to live together in a community where they have scheduled study sessions and activities. A dorm like this will have rules of conduct including quiet hours, visitation restrictions, and substance (read alcohol) bans. Even in my less-organized off-campus dorm we had weekly meetings, often ate meals together, had an RA in charge, participated in officially sanctioned events like intramurals and floor competitions, and had facilities on site for meetings, conferences, and parties.