Our stake, declared: we are Geonode and we sell proxies, which is the product most articles on this subject exist to sell. The honest ordering puts proxies about sixth, and the five things above them are free. Pacing, identification, caching, honouring Retry-After and reading robots.txt will prevent more blocks than any amount of address rotation, because they address the actual reason sites block crawlers — load and unpredictability — rather than the symptom. Proxies genuinely help with one specific problem, covered later. If you reach for them first, you will spend money and still get blocked.
Why Crawlers Actually Get Blocked
Four causes, in descending order of frequency.
Rate. You made too many requests too quickly. This is by far the most common, and it is entirely under your control. A site does not know or care who you are when it decides that thirty requests a second from one address is a problem.
Unpredictability. Bursts, retries into errors, crawling the same pages repeatedly, following infinite URL spaces. Load that a site cannot plan for is worse than load that it can.
Anonymity. An unidentified client generating unexplained traffic is a problem to be stopped. An identified one is a decision to be made, and frequently the decision is to allow it.
Identity signals. Address type, TLS fingerprint, header composition. Real, and last on this list because they matter mainly once a site has already decided it does not want unidentified automation — and because they are the hardest to change honestly.
Notice that three of the four are about behaviour. The industry's focus on the fourth is a function of what is easiest to sell, not of what causes most blocks.
Start by Not Needing to Crawl
The measure with the best return, and the one most often skipped.
Check for an API. Many sites publish one, and it is stable, structured and sanctioned. Crawling a site that offers an API is doing more work for a worse result.
Check for a partner or affiliate feed. Entire industries — jobs, property, retail, travel — publish bulk feeds specifically for aggregators, because aggregators send them traffic. Ask before you build. A surprising number of sites say yes.
Check for a sitemap. It gives you a URL inventory plus lastmod timestamps, so you fetch what changed rather than everything.
Check for embedded structured data. JSON-LD in a <script type="application/ld+json"> block is designed to be machine-read, survives redesigns, and is already in the page you were going to fetch.
Check whether the data exists elsewhere. Public datasets, archives, official filings.
Every one of these removes the blocking problem rather than mitigating it. The reflex to write a crawler first is the single most expensive habit in this field, and it is why we put this section before anything technical. We covered the full source hierarchy in how to find all pages on a website.
Read the Rules First
robots.txt is a standard now, RFC 9309, and honouring it correctly is both compliance and self-interest.
The parts that matter operationally: matching is by specificity rather than order — the longest matching rule wins; a 5xx response means complete disallow, not "carry on"; a 404 means no restrictions; and the file must be refreshed at least every 24 hours rather than fetched once at start-up.
Use a maintained parser. The specificity rule and percent-encoding normalisation are both easy to get wrong, and a crawler that misreads the file is one that believes it is compliant and is not. We went through the details in how to read a robots.txt file.
Then read the terms of service. robots.txt is not authorisation — the RFC says so directly — and a site may prohibit automated access regardless of what the file permits. Knowing that before you start is better than discovering it in a letter.
Pace Yourself Properly
The highest-value technical measure, and the cheapest.
One request every one to two seconds per domain is a reasonable default. Slower for small sites, and no faster without evidence that the target tolerates it.
Honour Crawl-delay where robots.txt sets one. It is an extension rather than part of the standard, and honouring it costs nothing and signals good faith.
Add jitter. Requests at exactly regular intervals are a signature no human produces. Randomising between, say, 1.0 and 2.5 seconds removes it at no cost.
Limit concurrency per domain, not globally. Eight concurrent requests spread across eight domains is polite. Eight against one domain is not.
Crawl during off-peak hours where you can. A site's tolerance is lower when it is busy, and a job that runs overnight costs you nothing extra.
And widen your window before adding capacity. This is the single most useful reframing available: fifty thousand pages spread over twenty-four hours needs about two concurrent connections; the same fifty thousand in two hours needs twenty. If nothing depends on the job finishing quickly — and usually nothing does — the schedule is the cheapest lever you have. We worked through the arithmetic in how many proxies do you need.
Identify Yourself
Counter-intuitive and consistently effective.
User-Agent: AcmePriceBot/1.2 (+https://acme.example.com/bot)
A name, a version, and a URL where someone can find out what you are and how to reach you. Three benefits, all real:
robots.txt can address you specifically. Rules match on the product token, so a site can grant your crawler an allowance it does not grant everyone. That cannot happen if you are anonymous.
Operators can contact you instead of blocking you. This happens more often than people expect, and it is a far better outcome than discovering a block three weeks later.
It supports asking for access. "We are the crawler identified as AcmePriceBot; here is what we collect and why" is a conversation that can go somewhere.
The alternative — a copied Chrome string — creates a contradiction rather than a disguise, because a browser user agent on a connection whose TLS fingerprint and header set are plainly not a browser's is more identifiable than an honest admission. We covered that in setting a custom user agent with curl.
Cache and Use Conditional Requests
The measure that reduces load without reducing coverage.
Never fetch the same unchanged resource twice. Store what you fetched, along with its ETag and Last-Modified values, then send conditional requests:
curl -sS -H 'If-None-Match: "abc123"' https://example.com/page
A 304 Not Modified costs a few hundred bytes instead of a full page. On a re-crawl where most pages are unchanged, this reduces both your bandwidth bill and the target's load by an order of magnitude.
Use lastmod from the sitemap to decide what to fetch at all. A site with fifty thousand pages of which two hundred changed today is a two-hundred-page crawl, not a fifty-thousand-page one.
Store raw responses. When a parser breaks, re-parse what you have rather than re-fetching. This is a cost saving and a courtesy at the same time.
Deduplicate URLs properly. Normalise trailing slashes, query parameter order, hostname case, and strip tracking parameters. A crawler without normalisation visits the same page many times and looks like a much heavier client than it is.
Handle Errors the Way the Server Asked
Servers tell you what to do. Reading the instruction is both correct and the fastest route back to working.
429 Too Many Requests is defined in RFC 6585 as indicating "that the user has sent too many requests in a given amount of time ('rate limiting')". The response "MAY include a Retry-After header indicating how long to wait before making a new request".
503 Service Unavailable means the server "is currently unable to handle the request due to a temporary overload or scheduled maintenance", and it "MAY send a Retry-After header field... to suggest an appropriate amount of time for the client to wait".
Retry-After takes either an HTTP date or a number of seconds, per RFC 9110 — Retry-After: 120 means wait two minutes.
The correct behaviour on a 429 or 503:
Stop for that domain. Not slow down, stop, for at least the interval given.
If no Retry-After is present, back off exponentially with a generous starting point.
Reduce your steady-state rate afterwards, because you have just been told it was too high.
Never retry immediately. Retrying into a rate limit is how a temporary restriction becomes a permanent block, and it is the most common self-inflicted injury in crawling.
Note also that RFC 6585 says "responses with the 429 status code MUST NOT be stored by a cache" — so a caching layer will not shield you from repeating the mistake.
Where Proxies Genuinely Help
Our own product, described as accurately as we can.
They help when: you have optimised your rate and still need throughput a single address cannot provide; you need to see region-specific content, where the whole point is to appear to be somewhere; you run distributed crawlers and want them to look like separate clients rather than one machine with many threads; or the address you are on has poor reputation through no fault of yours.
They do not help when: you are going too fast — the same rate from more addresses is the same rate, and now you have flagged a pool instead of an address. Nor when your request shape is being identified on TLS fingerprint or header composition, since those travel with you. Nor when a site has terms prohibiting automated access, which more addresses do not change.
Which type: datacentre for most crawling, because it is cheaper by a large factor and public pages usually do not require anything more — ours starts at $0.14/GB. Escalate to residential, from $0.79/GB, only where datacentre demonstrably fails or where you need consumer-network geolocation. Figures from our pricing page, checked September 2026.
The cost that surprises people: headless browsers. A browser fetches every image, font and script, so bandwidth goes up by roughly an order of magnitude over raw HTTP. If a page does not require JavaScript, do not render it — and if it does, block resource types you do not need.
Telling a Hard Block From a Soft One
The failure mode that costs most, because it does not look like a failure.
A hard block returns 403, a challenge page, or a connection refusal. Loud, obvious, and immediately actionable.
A soft block returns 200 with reduced content: fewer items, stripped fields, stale data, or a generic page instead of the specific one. Your success-rate metric stays at 99% and your data quietly degrades. This is the more common response from sophisticated sites, precisely because it wastes your budget without telling you anything.
Guard against it explicitly:
Assert content, not status. Check for a known-stable marker on the page and treat its absence as an error. Assert counts. If a category page has never had fewer than twenty items, make fewer than twenty a failure. Segment metrics by target. Twelve targets at 99% and one at 40% averages to something that reads as healthy. Compare against a browser periodically. Fetch one page manually and diff it against what your crawler received.
This is the same silent-failure pattern we described in why testing proxies matters, and it is the reason "we were blocked for three weeks and did not notice" is a real category of incident.
When to Stop
The section a proxy vendor has least incentive to write.
When the terms prohibit it. Some sites state this explicitly and enforce it. Engineering around a stated prohibition is a decision with consequences beyond the technical.
When you have been asked to stop. A direct request from a site operator ends the discussion.
When the effort exceeds the value. If you are rebuilding your approach every fortnight, the data is costing more than it is worth. That is a business conclusion, not a technical defeat.
When a legitimate route exists. An API, a feed, a licensed dataset. Paying for sanctioned access is frequently cheaper than the engineering time spent avoiding it, and it does not break.
When the site has deployed traps. Tarpits and generated mazes are designed to cost you more than they cost the site — they serve cached content while you pay per gigabyte and per compute-hour. That asymmetry is deliberate and it does not yield to effort.
Ask first. An email explaining who you are, what you need and at what volume resolves this more often than the discourse suggests, and it produces access that keeps working.
People Also Ask
Why does my crawler keep getting blocked?
Usually rate. Too many requests too quickly from one address is the most common cause by a wide margin, and it is entirely under your control. Unpredictable patterns, retrying into errors and anonymous identification account for most of the rest.
How fast can I crawl a website?
Start at one request every one to two seconds per domain and only go faster with evidence the target tolerates it. Honour Crawl-delay if robots.txt sets one. If you need more throughput, widening your time window is cheaper and safer than increasing concurrency.
Do proxies stop you getting blocked?
Only for address-specific blocks. If you are going too fast, the same rate from more addresses is the same rate and now flags a whole pool. If your request shape is identified on TLS fingerprint or headers, those travel with you regardless of address.
Should I rotate user agents?
No. Random rotation within a session produces a client that appears to change browser mid-visit, which is an inconsistency rather than a disguise. A single honest user agent with a contact URL gets blocked less often than any rotation scheme.
What should I do when I get a 429?
Stop for that domain and wait at least as long as Retry-After specifies. Without that header, back off exponentially from a generous starting point, then reduce your steady-state rate — you have just been told it was too high. Never retry immediately.
How do I know if I am being soft-blocked?
Assert on content rather than status codes. Check for a known-stable marker on each page, assert expected item counts, segment success metrics by target rather than in aggregate, and periodically compare a crawled page against one fetched in a browser.
Is crawling a website legal?
It depends on jurisdiction, the site's terms, what data is involved and what you do with it. Honouring robots.txt, identifying your crawler and keeping your rate modest is normal practice, and none of it overrides terms of service or copyright. Get advice for anything commercially significant.
What is the single most effective thing I can do?
Slow down, and check whether you need to crawl at all. An API, a partner feed or a sitemap with lastmod timestamps removes the problem rather than mitigating it — and where crawling is genuinely necessary, pacing prevents more blocks than every other measure combined.
Wrapping Up
The framing that makes this tractable is that blocking is a response to load and unpredictability, not to identity. Sites do not object to being read; they object to being hammered by something they cannot plan for and cannot contact.
Which puts the effective measures in an order most articles reverse. Check whether you need to crawl at all, because an API or a partner feed removes the problem entirely. Read robots.txt and honour it correctly, including the parts about specificity matching and 5xx meaning stop. Pace yourself and add jitter. Identify yourself with a name and a contact URL. Cache aggressively and use conditional requests so you never re-fetch what has not changed. Do what Retry-After tells you.
All of that is free, and it will prevent more blocks than any infrastructure purchase. Proxies help with a genuine and narrower problem — throughput past a single address's ceiling, and content that differs by region — and they help with nothing else on the list.
And keep the last section in view. A site that has stated terms, deployed traps or asked you to stop has told you something, and the alternatives to out-engineering it are usually cheaper and always more durable.