Our stake, declared: we are Geonode and we sell proxies, which is the infrastructure you would use to collect web data yourself. The honest position is that almost nobody should. The public corpora below represent enormous amounts of filtering, deduplication and legal care, they are free, and reproducing even a fraction of that work is a research programme rather than a project. Where collection is genuinely warranted — a narrow domain nobody has published, or fresh data past a corpus cut-off — it is a small, targeted job rather than a web-scale crawl. That section is at the end and it is short, deliberately.
The Layer Underneath: Common Crawl
Almost every open web corpus derives from the same source.
Common Crawl is a free web archive containing billions of pages, released as periodic crawl snapshots on AWS S3 in us-east-1 and available over HTTP at data.commoncrawl.org. Anonymous access works with the AWS CLI using --no-sign-request, or with ordinary tools such as wget and curl.
It publishes three formats, and knowing which you want saves considerable bandwidth:
WARC — "the raw data from the crawl, providing a direct mapping to the crawl process", including HTTP responses, requests and metadata. Complete and very large.
WAT — "Web Archive Transformation" files containing computed metadata as JSON, including HTTP headers and page links. The right choice for link-graph work.
WET — "WARC Encapsulated Text" files with extracted plaintext only. The right choice for language modelling, and dramatically smaller than WARC.
The important thing to understand is that Common Crawl is a raw ingredient, not a dataset. It contains boilerplate, navigation, spam, duplicates, machine translation and every other artefact of the open web. The value in the derived corpora below is almost entirely in the filtering, and that filtering is where the research is.
FineWeb
Hugging Face's web corpus, and the current reference point for open web data at scale.
FineWeb is derived from Common Crawl and contains 25.9 billion rows, covering crawls from CC-MAIN-2013-20 through CC-MAIN-2025-26 — roughly mid-2013 to mid-2025. It is released under ODC-BY, the Open Data Commons Attribution licence.
Each record carries quality signals including a language score and a token count, which matter more than they sound: they let you filter the corpus further for your own purposes without redoing the pipeline. Wanting only high-confidence English above a length threshold is a filter expression rather than a preprocessing job.
The reason FineWeb is worth starting with is that the filtering decisions are documented and ablated rather than asserted. When a corpus tells you which filtering choices improved downstream benchmark performance and by how much, you can disagree with them deliberately.
Dolma
Allen AI's corpus, and the most transparent about its composition.
Dolma is described as "a dataset of 3 trillion tokens from a diverse mix of web content, academic publications, code, books, and encyclopedic materials", comprising 2.532 billion documents. Version 1.7, containing 1.715 trillion tokens, was used to train OLMo 7B-v1.7.
Its sources are named individually: Common Crawl web pages, code from StarCoder and The Stack, academic papers from S2ORC and arXiv, Reddit, Project Gutenberg books, and Wikipedia.
It is released under ODC-BY, with an important caveat stated plainly: users "are also bound by the original licenses of constituent sources". That is the sentence to read twice. A permissive licence on the compilation does not override the licences of what went into it, and this is true of every derived corpus regardless of whether it says so as clearly.
Two further points make Dolma worth attention beyond its contents. Allen AI published the curation toolkit as open source, so the dataset is reproducible rather than merely downloadable — you can change a filtering decision and rebuild. And there is a removal mechanism: a form to notify the maintainers of documents containing personal information about a specific user.
That combination — named sources, open tooling, a removal path — is what responsible dataset publication looks like, and it is a reasonable standard to hold others to.
The Stack v2
The code corpus, and the most careful of the major datasets about licensing.
The Stack v2 from the BigCode Project is "a collection of source code in over 600 programming languages", containing 3.28 billion unique files totalling 67.53 TB uncompressed, spanning 658 languages. Training variants are filtered to either 17 or 600-plus languages depending on the version.
Its provenance is unusual and worth noting: the data derives from the Software Heritage archive, described as "the largest public archive of software source code", combined with GitHub Archive metadata through September 2023.
Two mechanisms distinguish it.
Licence filtering. The dataset "contains only permissively licensed code". The creators "propagate the detected licenses to all files" within repositories and maintain a curated list of acceptable SPDX identifiers based on Blue Oak Council approvals. That is a considerably more rigorous approach than filtering on a repository's declared licence field.
Developer opt-out. There is an "Am I In The Stack?" tool for checking inclusion, a documented removal process, and the dataset is "regularly updated to enact validated data removal requests".
Whatever one thinks of training on public code, this is the most defensible implementation of it currently available, and the opt-out mechanism is a genuine one rather than a gesture.
Licences and Provenance
The part that determines whether you can actually use any of this, and it has more layers than a single licence field suggests.
The compilation licence is not the content licence. ODC-BY on FineWeb or Dolma governs the collection. The underlying documents carry their own copyright, and Dolma says so explicitly. A permissive dataset licence means the compilers are not restricting you further; it does not mean the contents were theirs to relicense.
Attribution is a requirement, not a courtesy. ODC-BY is an attribution licence. If you use these datasets, attribute them.
Opt-outs are becoming standard and are worth honouring. The Stack's removal process and Dolma's personal-information form both exist because publishing at this scale creates obligations. Using a dataset means inheriting the version you downloaded — so re-pulling periodically is how removals actually take effect in your copy.
Personal data has its own regime. Web-scale corpora contain personal information, and in jurisdictions with data protection law that creates obligations for you as a processor, independently of any dataset licence. "It was in a public dataset" is not a lawful basis.
And the legal environment is unsettled. Whether training on copyrighted material is permitted, and under what conditions, is actively litigated in several jurisdictions. In the EU, the text and data mining framework contemplates machine-readable rights reservations, and the mechanisms for expressing them are still settling. Anyone building a product on this should be watching it rather than assuming today's position is final.
Evaluating a Dataset Before You Use It
A corpus is not a black box, and half an hour of inspection before committing storage and compute will tell you more than any model card summary.
Sample and read. Pull a few hundred documents at random and actually read them. This sounds unserious and it is the single most informative thing you can do. You will find out within minutes whether the boilerplate removal worked, how much machine-translated text is present, and whether the domain mix matches what the description claims.
Check the language distribution against your needs. A corpus described as multilingual may be 90% English with a long tail, which is fine if you want English and useless if you want Portuguese. Where quality signals such as a language score are provided — FineWeb includes one — use them rather than trusting the headline.
Measure duplication yourself. Even deduplicated corpora contain near-duplicates, and the rate varies by source. Hash a sample and count collisions; if the rate is high, you are training on the same content repeatedly and your effective dataset is smaller than the token count suggests.
Check the cut-off date. Every corpus has one, and it determines what the resulting model cannot know. FineWeb's crawls run to mid-2025; anything more recent is absent. For a domain that moves quickly, this may be disqualifying regardless of everything else.
Look for contamination against your evaluation set. If your benchmark questions and answers appear in the training corpus, your evaluation is measuring memorisation. This is common, it is easy to check with substring search on a sample, and it invalidates results in a way that is embarrassing to discover after publication.
And check the storage and bandwidth cost before downloading. The Stack v2 is 67.53 TB uncompressed. Multi-terabyte downloads have real costs in transfer, storage and time, and streaming a subset directly from object storage is frequently a better plan than pulling the whole thing to find out you wanted a tenth of it.
Do You Actually Need a Training Dataset?
The question worth asking before any of the above.
For pretraining a model from scratch, yes — and this is a research-scale undertaking. Compute in the hundreds of thousands of GPU-hours, a team who have done it before, and a reason why an existing open-weight model will not serve. Very few organisations are in this position, and the ones who are already know it.
For fine-tuning, you need something entirely different: a modest, high-quality, task-specific dataset. Thousands of examples rather than trillions of tokens, and the work is in curation rather than in scale. None of the corpora above is the right input.
For retrieval, you need your own documents indexed, not a training corpus at all. This is the case an enormous share of "we need training data" enquiries turn out to be, and it is answered by a vector index over content you already have.
For evaluation, you need a held-out set representative of your actual task, built by hand and small.
The failure mode this section exists to prevent is downloading a multi-terabyte corpus for a problem that needed two hundred curated examples. It happens, and the wasted effort is considerable.
Building Your Own, Honestly
If you have established that you need domain data nobody has published, here is what it actually involves — and where our own product fits.
Collection is the easy part and the smallest part. Fetching pages is a solved problem. Wherever possible, prefer sanctioned routes: APIs, bulk exports, partner feeds, existing archives. Crawling is the last resort, not the first step, and it comes with obligations — robots.txt, terms of service, copyright, database rights, and data protection law where personal data is involved.
Cleaning is most of the work. Boilerplate removal, language identification, quality filtering, near-duplicate detection at scale, personal-information detection and removal. The published corpora represent enormous effort here, and Dolma's open toolkit is worth studying before writing your own — reusing a documented pipeline is far better than reinventing one badly.
Deduplication deserves particular attention. Near-duplicates degrade training and inflate your apparent data volume. Doing it properly at scale requires MinHash or a similar technique, and it is the step most likely to be skipped and most likely to matter.
Documentation is not optional. Record why the dataset exists, what it contains, how and when it was collected, what is missing, and what it should not be used for. This is the difference between an asset and a liability, and it is close to impossible to reconstruct later.
Where proxies come in: collection at volume from many sources, or where content differs by region. Datacentre bandwidth is the sensible default — ours starts at $0.14/GB — with residential from $0.79/GB only where it is demonstrably needed, from our pricing page, checked September 2026.
And where they do not: if the data you need is in a published corpus, buying bandwidth to recreate it is straightforwardly wasteful. Check first. The corpora above cover an enormous amount of ground, and the filtering work embedded in them is worth more than the raw text.
People Also Ask
What datasets are used to train LLMs?
Most open models train on web corpora derived from Common Crawl — FineWeb and Dolma are the current reference points — mixed with code from The Stack, academic papers, books and encyclopedic content. Dolma names its sources individually, which is unusual and useful.
Is Common Crawl free to use?
The data is freely accessible on AWS S3 and over HTTP, with anonymous access supported. It publishes WARC, WAT and WET formats, and its terms of use apply. Note that free access to a web archive does not resolve the copyright status of the pages within it.
What licence are the major LLM datasets under?
FineWeb and Dolma are ODC-BY, the Open Data Commons Attribution licence. Dolma states explicitly that users are also bound by the original licences of the constituent sources — the compilation licence does not override the copyright of the underlying documents.
Can I remove my data from a training dataset?
Sometimes. The Stack v2 provides an "Am I In The Stack?" tool and a documented removal process, and is updated to enact validated removal requests. Dolma offers a form for reporting documents containing personal information. Mechanisms vary by dataset and are not universal.
How big are LLM training datasets?
Dolma is 3 trillion tokens across 2.532 billion documents. FineWeb contains 25.9 billion rows covering Common Crawl from 2013 to 2025. The Stack v2 has 3.28 billion files totalling 67.53 TB uncompressed across 658 programming languages.
Do I need a training dataset to fine-tune a model?
No — you need a small, high-quality, task-specific dataset, typically thousands of examples rather than trillions of tokens. The large pretraining corpora are the wrong input entirely, and the work in fine-tuning is curation rather than scale.
Should I build my own training dataset?
Only for domain data nobody has published. The public corpora embody enormous filtering and deduplication effort that is difficult to reproduce, and most requirements labelled "training data" turn out to be retrieval or fine-tuning problems needing far less. Check the existing corpora first.
What is the hardest part of building a dataset?
Cleaning and deduplication, not collection. Boilerplate removal, language identification, quality filtering, near-duplicate detection at scale and personal-information handling account for nearly all the effort. Allen AI's open Dolma toolkit is worth studying before writing your own pipeline.
Wrapping Up
The public LLM datasets are a remarkable resource: trillions of tokens, documented filtering, permissive compilation licences, and in the better cases open tooling and working opt-out mechanisms. FineWeb for web text, Dolma for a documented mixture, The Stack v2 for code, all built on Common Crawl or Software Heritage underneath.
Two things are worth carrying away about using them. The compilation licence is not the content licence — Dolma says so directly and it is true of all of them — so a permissive dataset licence is the beginning of your legal analysis rather than the end. And opt-out mechanisms only work if you re-pull, since your downloaded copy is frozen at the moment you took it.
The more useful conclusion is about scope. Most requirements described as needing training data turn out to be retrieval problems, answered by indexing documents you already have, or fine-tuning problems, answered by a few thousand carefully chosen examples. Both are far smaller and far more tractable than anything on this page.
And if you genuinely do need to collect data nobody has published, the collection is the easy part. The filtering, deduplication and documentation are the work — which is exactly why the published corpora are worth so much more than the raw text they contain.