0

I am reading this page, and I found this sentence that sounds very weird to me:

rotate your user agent from a pool of well-known ones from browsers (google around to get a list of them)

Here is the full context:

Some websites implement certain measures to prevent bots from crawling them, with varying degrees of sophistication. Getting around those measures can be difficult and tricky, and may sometimes require special infrastructure. Please consider contacting commercial support if in doubt.

Here are some tips to keep in mind when dealing with these kind of sites:

  • rotate your user agent from a pool of well-known ones from browsers (google around to get a list of them)
  • disable cookies (see COOKIES_ENABLED) as some sites may use cookies to spot bot behaviour

What does this mean?

6

When a browser makes a request to a web server, it will identify itself with something called "User-Agent". Each browser (actually each version of each browser) has its own User-Agent value. For example, according to Wikipedia, Safari on the iPad has used this following User-Agent value,

Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405

User-Agent is useful for web developers, because it lets them know who are their visitors; so that they can customize their websites and the user experience accordingly. However, it's quite common that your browser will allow you to change this User-Agent value.

Back to your sentence.

rotate your user agent from a pool of well-known ones from browsers (google around to get a list of them)

It means that the mentioned software will have a list of well-known User-Agents from various browsers. (It seems like you need to enter these User-Agent values yourself, as the part "google around to get a list of them" suggests.) They call this list a "pool". And the software will rotate the User-Agent value for you, using the values in this list (or pool).

The "rotate" here means that the software will use the first User-Agent value in the list first, then the second one in the list, then the third one, ... After it uses the last User-Agent value in the list, it will go back to use the first User-Agent value again. After that, the second one, the third one, ... And it will keep going on like that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.