Our stake: we are Geonode and we sell proxies, which are an input to crawling and not to list management. The honest observation is that a badly managed crawl list costs far more than a badly chosen proxy plan. A crawler without URL normalisation visits the same page dozens of times under different query-parameter orderings, and on per-gigabyte pricing you pay for every one of them. A crawler without budgets follows a calendar into infinity and bills you for it. Fixing the list is free; fixing it after the invoice is not. That section is below, and it is the one worth reading first.
What a Crawl List Actually Is
Three things, often conflated.
The seed list — where you start. A handful of URLs, or a full sitemap.
The frontier — URLs discovered and queued but not yet fetched. This is the working data structure and the one with all the design decisions in it.
The visited set — URLs already fetched, kept so you do not fetch them again.
The lifecycle is a loop: take a URL from the frontier, fetch it, extract links, normalise them, discard the ones already in the visited set or already queued, add the rest to the frontier, mark the URL visited. Repeat until the frontier is empty or a budget is exhausted.
Simple in outline. Every step has a detail that will cost you a day if you get it wrong.
Seeding: Where the First URLs Come From
In order of how much work each saves.
The sitemap. The best seed available, because it is the site's own inventory and includes lastmod timestamps telling you what changed. Find it via the Sitemap: directive in robots.txt, and follow sitemap index files recursively.
A partner feed or API. If one exists, you may not need a crawl at all.
Web archives. The Internet Archive's CDX API returns historical URLs for a domain, including pages no longer linked from anywhere. It costs the target nothing and finds orphans that crawling cannot.
Search operators. site: queries surface indexed pages and, more usefully, subdomains you did not know about.
Category and index pages. For a targeted crawl, seeding from the specific listing pages you care about is far more efficient than starting at the homepage and hoping.
The homepage. The last resort, and the default people reach for first. Starting from one URL and discovering everything by link traversal is the slowest route to the worst coverage.
We went through the full source hierarchy in how to find all pages on a website. The short version: a good seed list turns a crawl into a fetch.
URL Normalisation: The Step Everyone Skips
The single highest-value piece of engineering in a crawler, and the most commonly omitted.
Without it, these are five different URLs to your visited set and one page to the server:
http://Example.com/Products
http://example.com/products
http://example.com/products/
http://example.com:80/products
http://example.com/products?utm_source=email
RFC 3986 defines the normalisations that are always safe.
Case normalisation. "The scheme and host are case-insensitive and therefore should be normalized to lowercase. For example, the URI <HTTP://www.EXAMPLE.com/> is equivalent to <http://www.example.com/>." Note the limit: "The other generic syntax components are assumed to be case-sensitive unless specifically defined otherwise by the scheme." Paths are case-sensitive — do not lowercase them.
Also: "the hexadecimal digits within a percent-encoding triplet (e.g., %3a versus %3A) are case-insensitive and therefore should be normalized to use uppercase letters".
Percent-encoding normalisation. The RFC calls this "a frequent source of variance among otherwise identical URIs", since "some URI producers percent-encode octets that do not require percent-encoding". These "should be normalized by decoding any percent-encoded octet that corresponds to an unreserved character".
Path segment normalisation. Remove . and .. segments by applying the remove_dot_segments algorithm, because "some deployed implementations incorrectly assume that reference resolution is not necessary when the reference is already a URI".
Scheme-based normalisation. The RFC gives the canonical example — these four are equivalent:
http://example.com
http://example.com/
http://example.com:/
http://example.com:80/
So an empty path "should be normalized to a path of /", and a default or empty port "should be removed by scheme-based normalization".
Beyond the specification, three normalisations are pragmatic rather than strictly safe, and worth applying with care:
Strip tracking parameters. utm_*, fbclid, gclid, session identifiers. These almost never change the content and they multiply your URL count enormously. Keep a list rather than guessing.
Sort remaining query parameters. ?a=1&b=2 and ?b=2&a=1 are usually the same page. Usually — some applications are order-sensitive, so test on a sample.
Remove fragments. #section is client-side and never reaches the server. Always safe to drop for crawling purposes.
And respect rel="canonical". If a page declares a canonical URL, that is the site telling you which of several addresses is the real one. Honouring it is free deduplication with the site's own authority behind it.
The Frontier: Queue Design
Three properties decide whether your crawler scales.
Deduplication must be cheap. Before adding a URL, you check whether it is already known. At a million URLs a linear scan is unusable. A hash set works to a point; beyond that, a Bloom filter gives you constant-time membership at a fraction of the memory, with a small false-positive rate — meaning you occasionally skip a URL you have not seen. For most crawls that trade is fine; where completeness matters, back the filter with an exact store.
Order must be controllable. A plain FIFO queue gives breadth-first traversal, which is usually what you want — it reaches broad coverage early and stays shallow. A LIFO stack gives depth-first, which reaches deep into one branch and is rarely useful for a site crawl. A priority queue lets you order by anything you like, covered next.
Per-host state must be tracked. The frontier is not one queue in practice; it is a queue per host, so that politeness limits apply to each independently. A single global queue with a global rate limit means one large site starves every other.
The structure that scales is a set of per-host queues plus a scheduler that picks the next host eligible to be fetched from — eligible meaning enough time has elapsed since the last request to it.
Prioritisation
Which URL to fetch next, when you cannot fetch them all.
By depth. Shallower pages are usually more important. A simple and effective default.
By path pattern. If you want product pages, prioritise URLs matching /product/. This is the highest-return heuristic for targeted crawls and it is trivially cheap.
By lastmod. From the sitemap. Fetch what changed.
By change history. For recurring crawls, pages that changed often before will change often again.
By inbound link count. Pages linked from many places are usually more significant. Expensive to compute during a crawl and worth it for large jobs.
By estimated value. Whatever your actual objective is. If you want prices, prioritise pages likely to contain them.
The practical arrangement is a small integer score computed at enqueue time from a couple of these signals, used as the key in a priority queue. Elaborate schemes rarely justify their complexity; depth plus a path-pattern bonus covers most needs.
Bounding: Budgets and Traps
Without limits, some crawls do not terminate. This is not an edge case.
Maximum depth. Links from links from links. A depth cap of five or six covers almost any real site structure.
Maximum pages per host. A hard number. When it is reached, stop and report rather than continuing.
Maximum total pages. For the whole job.
Maximum bandwidth. Especially on metered proxy traffic, where an unbounded crawl is an unbounded invoice.
Pattern exclusions. Calendars are the classic infinite space — a next month link generates URLs forever. Faceted navigation on a large catalogue produces combinatorial explosions. Exclude these by pattern:
/calendar/
/?filter=
/*?sort=
Duplicate content detection. Hash the body. If a hundred URLs return identical content, you have found a generated space rather than a hundred pages.
And watch the discovery ratio. The most useful single alarm: track newly discovered URLs per page fetched. On a finite site this falls steadily towards zero. If it stays flat or rises, something is generating URLs faster than you can consume them — a trap, a calendar, or a faceted explosion. We covered the deliberate versions in honeypot traps.
Recrawl Scheduling
For anything running more than once, the list becomes a schedule.
Tier by volatility. Pages that change hourly need hourly checks; pages that changed once in a year do not. Crawling everything at the frequency the most volatile item requires is the single most common way to overspend.
Use conditional requests. If-Modified-Since and If-None-Match turn a re-crawl into a series of 304 Not Modified responses costing a few hundred bytes each. On a re-crawl where most pages are unchanged, this reduces both your bill and the target's load by an order of magnitude.
Adapt from observation. If a page has not changed in ten checks, check it less often. If it changed in the last three, check it more. A simple multiplicative back-off is enough.
Detect removal explicitly. Pages disappear, and sites rarely announce it. Either watch for 404s or apply a policy — a URL unseen in three consecutive crawls is marked inactive. Without this, your dataset fills with entries that no longer exist, which erodes trust faster than missing entries do.
Storage and Scale
A note on when the in-memory approach stops working.
Up to roughly a hundred thousand URLs, Python sets and lists are fine. Do not over-engineer.
Up to a few million, a local database — SQLite works well — with indexes on the URL hash and the fetch status. Persistence also means a crashed crawl resumes rather than restarts, which matters more than performance.
Beyond that, a proper queue and a key-value store, with the frontier partitioned by host so that workers can be assigned whole hosts and politeness stays correct without coordination.
Two design decisions that pay off at every scale:
Store the URL hash, not just the URL. Comparisons and indexes on a fixed-length hash are cheaper and the storage is smaller.
Store the raw response, not just the parsed result. When a parser breaks — and it will — re-parsing what you have is free, while re-fetching costs bandwidth and goodwill.
A Minimal Implementation
The whole thing in about forty lines, to make the moving parts concrete.
import time
from collections import deque, defaultdict
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode
TRACKING = {"utm_source", "utm_medium", "utm_campaign", "fbclid", "gclid"}
def normalise(url):
p = urlsplit(url)
host = p.hostname or ""
port = "" if p.port in (None, 80, 443) else f":{p.port}"
query = urlencode(sorted(
(k, v) for k, v in parse_qsl(p.query, keep_blank_values=True)
if k.lower() not in TRACKING
))
return urlunsplit((p.scheme.lower(), host + port, p.path or "/", query, ""))
class Frontier:
def __init__(self, delay=1.5, max_per_host=5000):
self.queues = defaultdict(deque)
self.seen = set()
self.next_ok = defaultdict(float)
self.counts = defaultdict(int)
self.delay, self.max_per_host = delay, max_per_host
def add(self, url, depth=0):
url = normalise(url)
if url in self.seen or depth > 5:
return False
host = urlsplit(url).hostname
if self.counts[host] >= self.max_per_host:
return False
self.seen.add(url)
self.counts[host] += 1
self.queues[host].append((url, depth))
return True
def next(self):
now = time.monotonic()
for host, q in self.queues.items():
if q and self.next_ok[host] <= now:
self.next_ok[host] = now + self.delay
return q.popleft()
return None
Five design decisions are visible in that, and each corresponds to a section above.
Normalisation happens at add(), not at fetch time. Deduplication is only correct if the canonical form is what enters the visited set, so normalising later means you have already stored duplicates.
The visited set is populated on enqueue, not on completion. Otherwise a URL discovered on twenty pages is queued twenty times before the first fetch finishes.
Queues are per host and politeness is per host. next_ok records when each host may next be contacted, so one large site cannot starve the others and the delay applies where it should.
Budgets are enforced at the point of entry. Depth and per-host caps reject URLs before they consume memory, which is the difference between a bounded crawl and one that discovers its limit by exhausting RAM.
next() returns None rather than blocking. That leaves the caller free to decide whether to wait, do other work, or finish — a scheduler that sleeps inside the data structure is one you cannot instrument.
What this deliberately lacks is persistence, and that is the first thing to add for anything real. A crashed crawl that must restart from the seed list has lost more than time; it has re-fetched everything, which costs bandwidth and goodwill.
Instrumenting the List
What to measure, because a crawl that reports only "pages fetched" tells you almost nothing.
Frontier size over time. Should fall towards zero. Rising means unbounded discovery.
Discovery ratio. New URLs per page fetched, as above.
Fetch outcomes by status, per host. Aggregate numbers hide one host failing entirely.
Duplicate rate. How many discovered URLs were already known. A high rate after normalisation means your normalisation is missing something.
Bytes per useful page. The number that connects the crawl to the invoice, and the one that reveals a headless browser fetching megabytes of images you did not need.
Content assertions. Whether pages contain the markers you expect. A crawler reporting 100% success while returning soft-blocked pages is the expensive failure, and only content assertions catch it.
People Also Ask
What is a crawl list?
The set of URLs a crawler intends to visit, usually comprising a seed list, a frontier of discovered-but-unfetched URLs, and a visited set. Managing the frontier — order, deduplication and budgets — is most of what determines whether a crawl finishes.
How do I normalise URLs for crawling?
Lowercase the scheme and host but not the path, uppercase percent-encoding hex digits, decode unnecessary percent-encoding, remove dot segments, drop default ports, normalise an empty path to /, remove fragments, and strip known tracking parameters. RFC 3986 defines all but the last two.
What is a crawl frontier?
The queue of discovered URLs not yet fetched. In practice it is a set of per-host queues plus a scheduler that picks the next host eligible under politeness limits, since a single global queue lets one large site starve every other.
How do I stop my crawler running forever?
Hard budgets: maximum depth, maximum pages per host and overall, and a bandwidth cap. Add pattern exclusions for calendars and faceted navigation, and alert when the ratio of newly discovered URLs to pages fetched stops declining.
How do I avoid crawling the same page twice?
Normalise URLs before deduplicating, since the same page has many valid addresses. Then check membership in a visited set — a hash set at small scale, a Bloom filter or a database at large scale. Honour rel="canonical" where pages declare it.
How often should I recrawl?
As rarely as your requirement tolerates, tiered by how often each page actually changes. Use lastmod from the sitemap and conditional requests so unchanged pages cost a 304 rather than a full fetch. Crawling everything at the volatile pages' frequency is the most common source of waste.
What is a Bloom filter and do I need one?
A probabilistic set that answers membership in constant time using very little memory, with a small chance of false positives — meaning you occasionally skip a URL you have not actually seen. Worth it beyond a few million URLs; unnecessary below that, where a plain set is simpler and exact.
How do I detect that pages have been removed?
Watch for 404s, and apply a policy for pages that simply stop appearing — for example, mark a URL inactive if it has not been seen in three consecutive crawls. Without this, a dataset accumulates entries that no longer exist, which damages trust faster than gaps do.
Wrapping Up
Crawling is mostly bookkeeping. The fetching is a solved problem with libraries; deciding what to fetch, recognising that you have already fetched it, and knowing when to stop are where the engineering is.
Three things repay attention out of proportion to their difficulty. Normalisation, because without it you visit the same page under a dozen addresses and pay for every one — and RFC 3986 tells you exactly which transformations are safe. Budgets, because some URL spaces are genuinely infinite and a crawler without hard limits will find one. And the discovery ratio, because it is a single number that reveals a trap, a calendar or a faceted explosion long before the invoice does.
Then seed well. A sitemap turns a crawl into a fetch of what changed, an archive query surfaces pages that link traversal can never reach, and both cost the target nothing. Starting at the homepage and hoping is the slowest route to the least complete result, and it is still the default.