Web Mine Tech — Web scraping and polite crawling | webminetech.com
Every response code is an instruction: 200 says proceed, 301 and 302 say follow and record, 403 says stop, 404 says skip, and 429 or 503 say wait.
A 200 OK is permission to parse, not proof of success. Sites under pressure sometimes return 200 with an empty shell page — a soft block — so verify that the expected table or element is present before writing a row. If the payload is missing, compare against a real browser fetch before assuming the layout changed.
301 Moved Permanently and 302 Found both tell the client to look elsewhere, and a scraper must follow both. The difference matters for bookkeeping: a 301 means update your stored URL because the move is forever, while a 302 means keep the original and expect the detour to change. Either way, record the final URL of every row so each record can be re-verified later.
403 Forbidden is the code that ends crawls. A single 403 on one path may be a protected directory; a 403 on every path, including the home page, means your agent or address has been refused. The correct response is to stop, re-read robots.txt, and lower your profile — not to rotate identities and push through.
On narrow screens, swipe or scroll the plate sideways.
404 Not Found is routine housekeeping: log it, skip the row, and do not retry it five times in a loop. A sudden wave of 404s across previously working URLs usually means the site restructured — re-check the sitemap before rewriting any selectors.
429 Too Many Requests, defined in RFC 6585 in 2012, is the server naming your exact problem: rate. It usually arrives with a Retry-After header, given either as a number of seconds or as an HTTP-date, and honoring that timer is the fastest route back to 200. The companion page on crawl rate and backoff walks through the doubling sequence for refusals without a header.
503 Service Unavailable means the server itself is overloaded or down for maintenance, not that you are banned. Treat it like a 429 with longer patience: back off, and if 503s persist for hours, park the job and resume the next day rather than hammering a struggling host.
Further reading