Website Ripper Tools: A Complete Guide

Our stake, declared: we are Geonode and we sell proxies, which are almost never needed for this. Mirroring a site you have permission to mirror is a single-source, moderate-volume job that runs fine from your own connection. If you are considering proxies for it, that usually means you are mirroring something at a rate the site objects to, and the correct fix is to slow down rather than to distribute. There is one genuine exception — mirroring region-specific content — and it is noted below. Everything else in this article works from a laptop with no infrastructure at all.

What These Tools Actually Do

Four steps, in every tool in the category.

Fetch a page. Download the HTML. Find the assets. Parse for images, stylesheets, scripts, fonts, and follow those too. Follow the links. Discover more pages within the scope you defined. Rewrite the references. Change absolute URLs to relative paths so the copy works from your filesystem.

That fourth step is what distinguishes a ripper from a crawler. A crawler collects data; a ripper produces a browsable local copy, and link rewriting is the difference.

The legitimate uses are unremarkable: archiving a site before it goes offline, taking documentation offline for travel or a restricted network, migrating between platforms, keeping a compliance record, and preserving your own work when a host shuts down.

wget: The Default Answer

Already on most Unix systems, and adequate for a large share of jobs.

wget --mirror --convert-links --adjust-extension --page-requisites \
     --no-parent --wait=1 --random-wait \
     https://example.com/docs/

Each flag earns its place, and the wget manual is precise about them.

--mirror "turns on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. It is currently equivalent to -r -N -l inf --no-remove-listing."

--page-requisites "causes Wget to download all the files that are necessary to properly display a given HTML page. This includes such things as inlined images, sounds, and referenced stylesheets." Without it you get HTML with no styling and no images.

--convert-links rewrites references "after the download is complete... to make them suitable for local viewing", affecting "not only the visible hyperlinks, but any part of the document that links to external content".

--adjust-extension appends .html to pages served as HTML without an HTML extension — the manual gives the example of "mirroring a remote site that uses .asp pages, but you want the mirrored pages to be viewable on your stock Apache server".

--no-parent ensures "that only the files below a certain hierarchy will be downloaded", which is what confines the job to /docs/ rather than the whole site.

--wait=1 is the courtesy flag, and the manual recommends it explicitly: "Use of this option is recommended, as it lightens the server load by making the requests less frequent." Pair it with --random-wait, which the manual notes was "inspired by this ill-advised recommendation to block many unrelated users from a web site due to the actions of one".

Two further options worth knowing. -Q sets a download quota so an unbounded mirror does not fill your disk. And -l sets a recursion depth if infinite is too generous.

wget's limitations are real: it does not run JavaScript, its link rewriting is good but not perfect on complex sites, and it has no graphical interface. For a documentation site or a static blog, none of that matters.

HTTrack: The Graphical Classic

HTTrack is the best-known dedicated tool in the category, open source, cross-platform, with both a graphical interface and a command line. The project remains live.

Where it beats wget: a genuine interface for people who do not live in a terminal, better handling of complex link structures, resumable projects, and an update mode that re-mirrors only what changed.

Where it does not: same fundamental limitation — no JavaScript execution — plus a filtering syntax that takes some learning, and a reputation among site operators that means some block it by user agent.

For a non-technical user who needs a browsable copy of a static site, this is the recommendation. For anyone comfortable with a shell, wget does the same job with fewer surprises.

Single-Page Tools

A different shape of problem, and frequently the right one.

If you want one page complete rather than a whole site, tools that inline everything into a single self-contained HTML file are more useful than a mirror. They embed images as data URIs, inline the CSS and produce one file you can email, archive or open anywhere with no dependencies.

Browser extensions in this family are the practical version for most people, and they have one decisive advantage over every command-line tool here: they capture the page as rendered, after JavaScript has run. For a modern application, that is the difference between a working copy and an empty shell.

The trade-off is that they are manual — one page at a time, with a human clicking. For a handful of pages that is fine; for a thousand it is not.

ArchiveBox and Preservation Tools

ArchiveBox is the notable option when the goal is preservation rather than offline browsing. It takes URLs and produces multiple archival formats at once — HTML, a screenshot, a PDF, extracted text, and a WARC file.

WARC is the format worth knowing about, because it is what web archives use. It stores the HTTP transactions themselves rather than a filesystem approximation, which means headers, status codes and the exact bytes are preserved. For anything where fidelity matters — legal, compliance, research — that is a materially better record than a directory of rewritten HTML.

ArchiveBox self-hosts, keeps an index, and handles JavaScript by driving a headless browser. It is heavier than wget and it produces something more durable.

Why They All Struggle With Modern Sites

The single explanation behind most disappointment in this category.

JavaScript rendering. wget and HTTrack fetch HTML and parse it. A single-page application returns a nearly empty document plus a script bundle, and the content is assembled in the browser. What you mirror is the shell.

API-driven content. Even where the initial HTML has content, subsequent navigation may fetch JSON from an API. Those requests are made by code, not by links in the markup, so a link-following mirror never discovers them.

Infinite scroll and lazy loading. Content that appears on interaction is not in the markup at all.

Client-side routing. URLs that never hit the server. A mirror cannot fetch what was never requested.

Authentication and personalisation. Anything behind a login, and anything that differs per user.

The workarounds, in order of effort:

Check for a static export. Documentation sites frequently offer a PDF or a downloadable bundle. Ask before building.

Check for an API. If the content comes from one, fetching it directly is easier and more complete than mirroring the front end.

Use a browser-based tool for the pages that genuinely need rendering. Playwright can navigate, wait for content and save the rendered HTML, which is a mirror with JavaScript support at the cost of writing a script and considerably more bandwidth.

Accept a partial mirror. For many purposes, the static parts of a site are what you wanted anyway.

Mirroring a JavaScript Site With Playwright

The practical fallback when the classic tools return an empty shell. Not a general-purpose ripper, but enough to capture a defined set of pages as rendered.

import { chromium } from 'playwright';
import { writeFile, mkdir } from 'fs/promises';
import { dirname } from 'path';

const urls = [/* the pages you want */];

const browser = await chromium.launch();
const ctx = await browser.newContext();
const page = await ctx.newPage();

for (const url of urls) {
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  await page.waitForSelector('main', { timeout: 15000 }).catch(() => {});

  const html = await page.content();          // rendered DOM, not source
  const path = 'mirror' + new URL(url).pathname.replace(/\/$/, '/index') + '.html';
  await mkdir(dirname(path), { recursive: true });
  await writeFile(path, html);

  await page.waitForTimeout(1000 + Math.random() * 1000);
}

await browser.close();

Four things in there matter.

page.content() returns the rendered DOM, not the source HTML. That is the entire reason this approach works where wget does not — you get the markup as it exists after JavaScript has built it.

Waiting for a selector, not for network idle. Pages with analytics beacons or websockets never go idle, so waitUntil: 'networkidle' will simply time out on many modern sites. Waiting for an element you know should be present is both faster and more reliable.

The deliberate pause between pages. Same courtesy as --wait in wget, and just as necessary.

No link rewriting. This is the honest limitation: you get rendered HTML with absolute references, so the copy needs an internet connection to look right. Adding rewriting means parsing each document, downloading every asset and rewriting the references — which is reimplementing wget, and at that point combining the two is easier: use Playwright to render and save, then run wget against the saved files to collect assets.

And expect the bandwidth cost. A browser fetches every image, font, script and video preload, so this consumes roughly an order of magnitude more traffic than a wget mirror of the same pages. Blocking font and media requests with page.route() cuts it substantially where those are not part of what you are preserving.

Choosing a Tool

That last row is worth stating on its own, because it is the case people most often solve the hard way. If you own the site, use the platform's export. A database dump, a static-site build or a hosting provider's backup is complete, includes what mirroring cannot see, and takes minutes. Mirroring your own site is doing the difficult version of an easy task.

The Rules That Apply Regardless

Independent of tool.

robots.txt applies to you. wget honours it by default. HTTrack honours it by default. Both can be told not to, and doing so is a deliberate choice with consequences. It is a standard now — RFC 9309 — and we covered reading it in how to read a robots.txt file.

Rate limiting is not optional. A mirror without --wait sends requests as fast as the connection allows, which is indistinguishable from an attack from the server's side. One request per second is a reasonable floor.

Copyright does not disappear. Downloading a copy for personal offline reading is one thing; republishing it is another. The content remains the owner's.

Terms of service may prohibit it outright, regardless of technical feasibility.

Bandwidth costs the site money. A mirror of a large site can transfer many gigabytes, and someone pays for that.

Ask. For anything substantial, an email is faster than the workaround. Site owners frequently say yes, and sometimes offer a bundle that saves you the exercise entirely.

Where Proxies Fit, Briefly

Since this is our product and the honest answer is short.

You do not need them for mirroring a site you have permission to mirror at a polite rate from one location. That is the overwhelming majority of legitimate use, and a single connection handles it.

You might need them if the site serves different content by region and you want the regional versions — a documentation site with localised pages, or a catalogue with country-specific inventory. Here the geography is the point, and it is a genuine use case.

You do not need them to go faster, and reaching for them for that reason means you are mirroring at a rate the site would object to. The correct response to that is --wait, not distribution.

If the regional case applies, datacentre bandwidth is the sensible choice — ours starts at $0.14/GB, checked September 2026 against our pricing page — and be aware that a full mirror is measured in gigabytes, so the arithmetic matters.

People Also Ask

What is a website ripper?

A tool that downloads a website's pages and assets to local storage and rewrites the links so the copy works offline. The link rewriting is what distinguishes it from a crawler, which collects data rather than producing a browsable mirror.

How do I download an entire website?

wget --mirror --convert-links --adjust-extension --page-requisites --no-parent --wait=1 URL covers most static sites. For a graphical alternative, HTTrack does the same job. For a JavaScript-heavy site, neither will work well and you need a browser-based approach.

Why does my downloaded website look broken?

Usually a missing --page-requisites, so stylesheets and images were not fetched, or --convert-links, so references still point at the live site. If the pages are empty rather than unstyled, the site renders content with JavaScript and a non-rendering tool cannot capture it.

Is downloading a website legal?

Downloading for personal offline use is generally unremarkable; republishing is a different question, since copyright still applies. Terms of service may prohibit automated downloading outright regardless of technical means. This varies by jurisdiction and is not legal advice.

Can I download a website that uses JavaScript?

Not with wget or HTTrack, which fetch and parse HTML without executing scripts. You need a browser-based approach — a single-file extension for individual pages, or a Playwright script that navigates, waits for content and saves the rendered result.

What is the best free website downloader?

wget if you are comfortable with a command line, since it is already installed and fully capable for static sites. HTTrack if you want a graphical interface. ArchiveBox if the goal is preservation rather than browsing, since it produces WARC files alongside HTML.

How do I download a website without getting blocked?

Set a delay of at least one second between requests, honour robots.txt, identify yourself honestly, and limit the scope with --no-parent and a depth limit. Most blocks in this category come from mirroring at full speed, which looks identical to an attack from the server's perspective.

Should I use a proxy to download a website?

Only if you need region-specific versions of the content. For an ordinary mirror at a polite rate, one connection is enough. Reaching for proxies to go faster means you are transferring at a rate the site objects to, and the fix for that is a delay flag.

Wrapping Up

For a static site, this is a solved problem and the tool is already on your machine. One wget command with five flags produces a browsable local copy, and the only flag people forget is the one that makes it polite.

For a modern application, none of the classic tools work, and the reason is not a shortcoming you can configure around. They fetch and parse HTML; the content is assembled by JavaScript after the fetch. The realistic options are a browser-based capture for the pages that matter, an API if one exists, or an export if you own the site — and that last one is the case most often solved the hard way.

Whatever you use, three things apply regardless of tool. Rate-limit, because a full-speed mirror is indistinguishable from an attack. Honour robots.txt, because it is a standard and ignoring it is a choice. And for anything substantial, ask — an email takes a minute and quite often produces a bundle that makes the whole exercise unnecessary.