Web Mine Tech — Web scraping and polite crawling | webminetech.com
Hold to one request per second, and when refused, double your wait: 1, 2, 4, 8, 16 seconds, capped at 30–60.
The polite default is one request per second — 86,400 requests a day from a single client, which outpaces most research needs. On shared hosting, where one machine serves many unrelated sites, the expected range tightens to one request every 2–10 seconds.
When a server answers 429, it usually attaches Retry-After. The header carries either a plain number of seconds or a full HTTP-date, and a correct client parses both forms. Waiting exactly as long as asked is faster, in total, than any clever retry pattern.
When no Retry-After is present, use exponential backoff: double the wait after each refused attempt — 1, 2, 4, 8, 16 seconds — and cap the growth at 30–60 seconds. Six failed attempts under this scheme span at most about a minute of waiting, after which the honest move is a cool-down of hours, not a seventh try.
On narrow screens, swipe or scroll the plate sideways.
Timeouts are rate signals too. Headless Chromium waits 30,000 milliseconds for a navigation by default, and plain HTTP clients are usually configured at 10–30 seconds; repeated stalls at either limit mean the server is shedding load, and the answer is a longer delay, not more parallel connections.
You can also shrink the load each request creates. Enabling gzip or brotli compression cuts HTML, CSS, and JSON payloads by 60–80%, which on a 1,000-page run is the difference between a noticeable footprint and a negligible one.
Sometimes the right rate is zero: if monthly-fresh data is enough, Common Crawl republishes roughly 2–3 billion pages per crawl, already fetched and stored. The robots.txt guide and the status codes reference cover the other two legs of a polite crawl.
Further reading