The use of the definite article in both those cases is optional but the use of the article tends to mean that the group referred to is more specific.
In the case of the nephew 'guests' without the definite article would mean anyone who was visiting socially whereas 'the guests' would either refer to people who were there for some more definite purpose, like a formal dinner, a wedding or a funeral or, if his parents were running an hotel or boarding house, 'the guests' would be any customers who were staying there or who had stayed there at any time. Similarly 'the neighbours' would probably mean a smaller group than 'neighbours', perhaps those living in the adjacent houses rather than the ones living several doors down.
In the case of the homes and population 'the homes' and 'the increasing population' would tend to be used when the homes and increasing population had been described previously and 'homes' and 'increasing population' when the discussion was more general.
Please note that these indications do not form rigid rules, there is a lot of choice in whether to use the definite article or not. Also when there is more than one group being discussed it is perfectly permissible to use the definite article with one and not the other.
For example 'My nephew would become anxious when asked to introduce himself to the guests and mingle with neighbours' suggests that he was living in an hotel and being asked to mingle not only with the people next door but also with people from the local area more generally.
Similarly 'The decision on where to build new homes to accommodate the increasing population' suggests that there has already been mention of an increase in population earlier in the piece of writing but that the topic of new homes is being introduced at that point.