The disclosure: we are Geonode and we sell proxies, which are one input to the crawling method described near the end. The honest ordering is that crawling should be the last thing you try, not the first. Five of the eight sources below are free, require no infrastructure, and frequently produce a more complete list than a crawl does — because they include pages that are no longer linked from anywhere. If you start by writing a crawler, you will do more work and get less coverage. Buy bandwidth when you have established that the free sources have gaps you need to fill, which is later than most people assume.
Why "All Pages" Has No Complete Answer
Four reasons, all structural.
Orphan pages exist. A page with no inbound links is unreachable by crawling and invisible to search engines, but it exists and may be live. Old landing pages, campaign URLs and deprecated sections all fall into this category.
Content behind forms and authentication is not enumerable. Search results, filtered views and anything requiring a login are not addressable by discovery.
Dynamic URLs can be infinite. A calendar with next-month links generates unbounded URLs. Faceted navigation on an e-commerce site produces combinatorial explosions. "All pages" is not a finite set on some sites.
Sites lie by omission. A sitemap contains what the owner chose to list, which is frequently the pages they want indexed rather than the pages that exist.
So the realistic goal is not completeness but sufficient coverage for your purpose, which means the sources you choose depend on why you are asking.
Start With the Sitemap
The single most productive first step, and the one people skip.
The sitemaps protocol defines an XML format listing a site's URLs. A sitemap "must begin with an opening <urlset> tag and end with a closing </urlset> tag", with each entry requiring a <loc> element. Optional <lastmod>, <changefreq> and <priority> elements may accompany it, though the protocol notes their treatment "may vary among search engines".
The limits matter for what you find: a single sitemap file is capped at "50,000 URLs and must be no larger than 50MB (52,428,800 bytes)". Larger sites use a sitemap index — a <sitemapindex> listing other sitemaps — which is subject to the same limits, so a site may have dozens of sitemap files.
Where to look:
https://example.com/sitemap.xml
https://example.com/sitemap_index.xml
https://example.com/sitemap.xml.gz
Gzip compression is permitted, "but the decompressed file must still comply with size restrictions".
And check robots.txt first, because the protocol provides for announcing sitemaps there with a Sitemap: directive:
curl -s https://example.com/robots.txt | grep -i sitemap
This is the highest-value thirty seconds available. Many sites list several sitemaps you would never have guessed the names of — separate ones for products, categories, blog posts and images.
Follow the index recursively. A sitemap index points at sitemaps which may point at further sitemaps. Fetch each, extract <loc> values, and note that entries in an index are sitemap files while entries in a urlset are pages.
The <lastmod> field is the other reason to start here: it tells you what changed, which turns a re-crawl into a fetch of the handful of pages that moved.
Read robots.txt for What It Reveals
Beyond the sitemap directive, robots.txt is an inventory of paths the owner considered worth mentioning.
Disallow: /admin/
Disallow: /internal/reports/
Disallow: /checkout/
Disallow: /search?
Every Disallow line names a path that exists. This is not a way in — it is a statement of what you should not crawl, and honouring it is both correct and the difference between an identified crawler and a nuisance. But it tells you the site's structure, and it frequently names sections you did not know about.
Read it as a map rather than a target list. A path that is disallowed should stay off your crawl list; knowing it exists still informs your understanding of the site.
Search Engine Operators
Fast, free, and partial.
site:example.com
site:example.com inurl:/products/
site:example.com -inurl:/blog/
site:example.com filetype:pdf
What this gives you: pages the search engine has indexed, which is a subset of pages that exist. Results are also capped and estimated rather than exhaustive.
What it is genuinely good for: discovering subdomains and sections you did not know about, and finding document files that are not linked from navigation. A filetype:pdf search on a corporate site routinely surfaces material nobody expected to be public.
The limitation is that scraping search results directly violates most search engines' terms, and the manual version is slow. If you need this programmatically, use an official search API where one exists rather than automating the web interface.
Web Archives
The source most people forget, and the only one that finds pages that no longer exist.
The Internet Archive's CDX Server API queries the archive's capture index directly. The documentation notes that "the most simple query and the only required param for the CDX server is the url param".
curl -s "http://web.archive.org/cdx/search/cdx?url=example.com/*&output=json&fl=original&collapse=urlkey&limit=10000"
The parameters worth knowing:
matchType controls scope — exact matches one URL, prefix returns everything under a path, host covers a single hostname, and domain covers a domain and all its subdomains. A wildcard in the URL sets this implicitly, so example.com/* is prefix matching.
collapse=urlkey removes adjacent duplicates, which is essential since the archive holds many captures of the same URL.
output=json is more convenient than the default CDX text format, and gzip=false turns off the default gzip encoding if your client cannot handle it.
Limits: the API "enforces a default maximum of 150,000 results per query", adjustable with limit=N, and for larger queries the documentation recommends the pagination API using page and pageSize.
Why this is uniquely valuable: it surfaces historical URLs. Pages that were removed, sections that were restructured, campaign landing pages that were unlinked. For understanding what a site used to be, or for finding orphaned content that is still live, nothing else compares — and it puts no load on the target at all.
Common Crawl
A large public crawl corpus with an index you can query, and a genuinely underused resource.
Common Crawl publishes periodic crawls of a substantial portion of the web, along with a URL index. Querying it gives you the URLs that crawl saw for a domain, in bulk, without touching the site.
The trade-offs are straightforward. Coverage is broad but not complete — it is a crawl, subject to the same blind spots as any crawl. Freshness depends on which crawl you query. And querying the index at scale is a data-processing exercise rather than a single request.
Where it earns its place: research at scale, comparing many domains, and any situation where you want a view of a site without generating traffic to it.
Crawling: The Last Resort, Done Properly
When the free sources leave gaps, crawl. Do it in a way that does not create problems.
The basic loop: fetch a page, extract links, filter to the target domain, queue the new ones, repeat until the queue empties. Simple in principle and full of details.
Details that matter:
Honour robots.txt. It is a standard now — RFC 9309 defines matching by specificity rather than order, requires refreshing the file at least daily, and treats a server error as complete disallow. Use a maintained library rather than writing the parser yourself.
Normalise URLs aggressively. Trailing slashes, query parameter order, case in hostnames, session identifiers and tracking parameters all produce duplicates. A crawler without normalisation will visit the same page hundreds of times.
Bound the crawl. Depth limits, page-count limits, and pattern exclusions for calendars and faceted navigation. Without them, some sites are infinite.
Rate-limit yourself. One request every second or two is polite and adequate for most jobs. Crawl-delay in robots.txt is a request worth honouring.
Identify yourself. A user agent with a name and a contact URL gets blocked far less readily than anonymous automation.
Cache and use conditional requests. If-Modified-Since and If-None-Match turn a re-crawl into a series of cheap 304s.
Where proxies come in: at genuine scale, when a single address gets rate-limited, or when the content differs by region. Not before. Datacentre bandwidth is the sensible default for this — ours starts at $0.14/GB, checked September 2026 — and residential is worth escalating to only when datacentre demonstrably fails.
Other Sources Worth Knowing
Four smaller ones that fill specific gaps.
Certificate transparency logs. Every TLS certificate issued is publicly logged, and certificates name their hostnames. Querying a CT log aggregator for a domain reveals subdomains — including internal-sounding ones that were never meant to be found by other means. This is the single best subdomain discovery method and it involves no contact with the target.
RSS and Atom feeds. Still widely published, and they list content in a structured form with dates. Check /feed, /rss, /atom.xml and the <link rel="alternate"> tags in the page head.
The site's own search and navigation. An A-to-Z index, a tag listing, a category page or an HTML sitemap page — many sites publish one for human visitors, and it is frequently more complete than the XML sitemap.
The API behind the front end. If the site is a single-page application, it is calling an API to get its content, and that API often exposes listing endpoints returning everything in a structured form. Look in your browser's network tab. This is consistently the fastest route on modern sites and consistently the one people find last.
Combining Sources
The practical method for a serious enumeration, and the reason the ordering matters.
Collect independently and merge. Sitemap URLs, archive URLs, feed URLs and crawl results into one set. Normalise before merging, or you will count the same page several times.
Verify liveness. An archive URL may 404 today. A HEAD request per URL is cheap:
while read -r url; do
code=$(curl -sIL -o /dev/null --max-time 10 -w '%{response_code}' "$url")
echo "$code $url"
done < urls.txt
Compare the sources against each other. URLs in the archive but not the sitemap are removed or orphaned pages. URLs in the sitemap but returning 404 are stale entries. URLs found by crawling but absent from the sitemap are pages the owner did not want indexed. Each difference is information.
Track over time. Re-running periodically and diffing tells you what was added and removed, which is frequently the actual question behind "find all pages".
A Practical Walkthrough
Putting the sources together on a single domain, in the order that does the least work.
One — find the declared sitemaps:
DOMAIN="example.com"
curl -s "https://$DOMAIN/robots.txt" | grep -i '^sitemap:' | awk '{print $2}' > sitemaps.txt
# fall back to the conventional locations if nothing is declared
[ -s sitemaps.txt ] || printf 'https://%s/sitemap.xml\nhttps://%s/sitemap_index.xml\n' "$DOMAIN" "$DOMAIN" > sitemaps.txt
Two — expand indexes and collect URLs. A sitemap index contains <loc> values pointing at other sitemaps, and a urlset contains <loc> values pointing at pages, so the same extraction works at both levels and you simply repeat it:
extract() { curl -s --compressed "$1" | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g'; }
: > all_urls.txt
while read -r sm; do
extract "$sm" | while read -r u; do
case "$u" in
*.xml|*.xml.gz) extract "$u" >> all_urls.txt ;;
*) echo "$u" >> all_urls.txt ;;
esac
done
done < sitemaps.txt
sort -u all_urls.txt -o all_urls.txt
wc -l all_urls.txt
Three — add the archive's view:
curl -s "http://web.archive.org/cdx/search/cdx?url=${DOMAIN}/*&output=text&fl=original&collapse=urlkey&limit=50000" \
| sort -u > archive_urls.txt
wc -l archive_urls.txt
Four — compare rather than merely combine. This is where the interesting output is:
comm -13 all_urls.txt archive_urls.txt > only_in_archive.txt # orphaned or removed
comm -23 all_urls.txt archive_urls.txt > only_in_sitemap.txt # new or never archived
only_in_archive.txt is the list worth looking at first. Those are URLs the site once served and no longer advertises — some will 404, and the ones that do not are live pages nobody links to.
Five — check liveness before trusting anything. Both lists contain stale entries, and a HEAD request per URL costs a few hundred bytes rather than a full page:
while read -r u; do
printf '%s %s\n' "$(curl -sIL -o /dev/null --max-time 10 -w '%{response_code}' "$u")" "$u"
done < only_in_archive.txt | tee liveness.txt
grep '^200 ' liveness.txt | wc -l
Note the deliberate ordering: four sources consulted, thousands of URLs collected, and not a single crawl written. On most sites this produces a more complete picture than a crawler would, in a fraction of the time, and the target barely notices you were there.
People Also Ask
How do I find all the pages on a website?
Start with robots.txt to find sitemap declarations, then fetch the sitemaps and follow any index files. Supplement with the Internet Archive's CDX API for historical URLs, RSS feeds, and a site: search. Crawl only for what those miss — it is the slowest and most intrusive option.
Where is a website's sitemap?
Usually /sitemap.xml or /sitemap_index.xml, and the reliable way to find it is the Sitemap: directive in robots.txt. Large sites use a sitemap index pointing at multiple files, since each is limited to 50,000 URLs and 50MB.
Can I find pages that are not linked anywhere?
Sometimes. Crawling cannot find them by definition, but web archives often can — the Internet Archive's CDX API returns historical URLs including ones no longer linked. Certificate transparency logs also reveal subdomains that are not linked from anywhere.
How do I find all subdomains of a website?
Certificate transparency logs are the best source, since every issued TLS certificate is publicly logged with its hostnames. Search engine operators and DNS enumeration tools supplement it. None of these requires contacting the target.
Is it legal to crawl a website to list its pages?
Depends on jurisdiction, the site's terms, and what you do with the result. Honouring robots.txt, identifying your crawler and keeping your rate modest keeps you within normal practice. Terms of service may prohibit automated access regardless, and that is a contractual matter worth checking.
How many URLs can a sitemap contain?
50,000 per file, with a 50MB uncompressed limit. Beyond that, sites use a sitemap index listing multiple sitemap files — and the index itself is subject to the same 50,000 and 50MB limits.
What is the Wayback CDX API?
An interface to the Internet Archive's capture index that returns the URLs it has archived for a domain. matchType controls scope from a single URL up to a domain and all subdomains, collapse=urlkey removes duplicate captures, and the default result cap is 150,000 with a pagination API for larger queries.
Do I need proxies to enumerate a website's pages?
Not for the sitemap, archive, feed or search approaches — none of them generate meaningful load. You need them for large crawls where a single address gets rate-limited, or where content differs by region. Establish that the free sources have gaps before buying anything.
Wrapping Up
There is no complete list of a website's pages, only a union of partial views — and the useful discipline is to gather the cheap views before building the expensive one.
Thirty seconds on robots.txt finds the sitemap declarations. A few minutes following the sitemap index gives you what the owner considers their site. The Internet Archive's CDX API adds the pages that used to exist and the ones nobody links to any more. Feeds, certificate transparency logs and the site's own HTML index each fill a different gap. All of it is free and none of it puts load on the target.
Crawl for what remains, with robots.txt honoured, URLs normalised, the crawl bounded, and your rate kept modest — and check first whether the site is a single-page application calling an API, because that route is usually faster than crawling and almost always overlooked.
Then compare the sources rather than merely merging them. The pages in the archive but not the sitemap, and the sitemap entries that now return 404, are often the most interesting thing the whole exercise produces.