This sentence isn't 100% clear as it is written. It needs at least one comma.
I think it should be:
"It is thought that the Neolithic settlers[comma], mingling with
peoples already present[comma], or taking over, came from the
continent and brought their own animals..."
In other words, Neolithic settlers came to England from the Continent. Either they mingled with the people already there, or they "took over" (displaced and/or eventually eliminated those other people), and they brought their own animals with them from the Continent.
Another possible meaning is that the Neolithic settlers came into England. There were other people either already in England or coming very soon to take over. The Neolithics brought their own animals when they mingled with those other people in England.
This interpretation only requires 1 comma:
"It is thought that the Neolithic settlers[comma], mingling with
peoples already present or taking over, came from the continent and
brought their own animals..."
Hard to be sure if you don't know the history, but either way, the point is a lot of the native animals disappeared because of the imported ones.