# Common Crawl vs curated legal data for LLM training

> Common Crawl is free and huge (over 300 billion pages, per its homepage in November 2026), but it holds legal text as page snapshots with menus, no article structure, no identifiers or versions, uneven coverage, and no licence of its own; many official legal portals also block CCBot or AI crawlers. For legal pre-training breadth it helps; for fine-tuning, RAG and evals, a curated corpus with structure, dates and per-source licences is what makes results verifiable.

Source: https://data.arietalabs.com/guides/common-crawl-vs-curated-legal-data/
Published: 2026-11-13 · Updated: 2026-11-14 · Author: Nejc Fošnarič, Founder, Arieta Labs (https://data.arietalabs.com/about/#nejc-fosnaric/)

Common Crawl is good for breadth and bad for law. It is a free archive of [over 300 billion web pages](https://commoncrawl.org/) (its homepage's figure in November 2026), run by a non-profit founded in 2007, and it underpins most open pre-training corpora. But it stores legal texts as web-page snapshots, with navigation and banners, no article structure, no stable identifiers, no version dates and no guarantee that a given act was crawled at all. It grants no licence for the content it holds, and several official legal portals block its crawler. For general pre-training, the legal text inside Common Crawl adds some domain knowledge. For fine-tuning, retrieval, evaluation and any use where answers must be cited and checked, a curated corpus built from official sources is the better input.

## What is Common Crawl?

[Common Crawl](https://commoncrawl.org/overview) is a 501(c)(3) non-profit, founded in 2007, that crawls the web and publishes the results for free. New crawls are released roughly monthly, each adding billions of pages, and the data sits on AWS as open data (`s3://commoncrawl/`). Each crawl comes in [three formats](https://commoncrawl.org/get-started): WARC files with the raw responses, WAT files with metadata, and WET files with extracted plain text.

Most open web corpora are filtered views of Common Crawl. C4 was introduced with T5 by [Raffel et al. (arXiv:1910.10683)](https://arxiv.org/abs/1910.10683). FineWeb, from [Penedo et al. (arXiv:2406.17557)](https://arxiv.org/abs/2406.17557), is a 15-trillion-token dataset derived from 96 Common Crawl snapshots. If your model was pre-trained on web data, it has probably seen some of the legal text that Common Crawl happened to capture.

## How much legal text does Common Crawl actually contain?

Common Crawl does not set out to cover law, and it does not say what share of any legal portal it holds. What it holds is whatever its crawler reached and was allowed to fetch. You can check a specific site through the [Common Crawl index](https://index.commoncrawl.org/), which lets you query captures by URL pattern or domain, and that is the first thing to do before relying on it for any jurisdiction.

Three limits apply to legal portals in particular:

| Limit | Why it matters for legal data |
|---|---|
| Crawler access | CCBot follows robots.txt, including `Crawl-delay`, and honours `nofollow` ([CCBot FAQ](https://commoncrawl.org/faq)). Portals that block it are simply missing |
| Page-level capture | A consolidated act on a portal is often split across pages, tabs or dynamically loaded views; a crawl captures pages, not acts |
| Sampling, not completeness | Each crawl covers part of the web. There is no guarantee that every act, every language version or every amendment is in any crawl |

Official legal portals increasingly opt out. The robots.txt of the Dutch statute portal [wetten.overheid.nl](https://wetten.overheid.nl/robots.txt), for example, disallows CCBot, GPTBot, ClaudeBot, Google-Extended and other AI crawlers, while allowing named legal-research bots under a crawl delay. The trend is general: [Longpre et al., "Consent in Crisis" (arXiv:2407.14933)](https://arxiv.org/abs/2407.14933) found that in 2023–2024 about 5 percent of all C4 tokens, and 28 percent or more of the most actively maintained, critical sources in C4, became fully restricted by robots.txt.

## What is wrong with web-crawled legal text?

Even where legal text is in a crawl, it arrives in a form that suits language modelling and little else.

| Property | Common Crawl / web corpora | Curated legal corpus |
|---|---|---|
| Unit | A web page | A document, split into recitals, articles and annexes |
| Boilerplate | Menus, cookie banners, footers, search boxes (WET text includes them unless filtered) | None: parsed text only |
| Identifiers | A URL, which changes when portals are redesigned | CELEX, ELI, ECLI or the national id |
| Dates | The crawl date | Date of document, entry into force, publication, and observation time per version |
| Versions | Whatever was online on crawl day, with no link between captures | A content-hash version history |
| Relations | Hyperlinks, if any | Typed amendment, repeal, legal-basis and transposition edges |
| Coverage | Unknown per jurisdiction | Stated per source, with counts |
| Licence | None granted; you inherit each page's terms | Recorded per record, with attribution text |
| Withdrawals | Old captures stay in old crawls | Withdrawn records delivered as [tombstones](/glossary/tombstone/) |

Filtering can introduce its own problems. [Dodge et al. (EMNLP 2021, arXiv:2104.08758)](https://arxiv.org/abs/2104.08758) documented C4 and found that its single most represented website was patents.google.com, and that blocklist filtering disproportionately removed text from and about minority individuals. Legal text, with its references to crimes, sexual offences and violence, is exactly the kind of content naive quality and safety filters remove, so a filtered web corpus can under-represent criminal and family law.

## What is the licence position?

Common Crawl licenses its own service, not the content. Its [terms of use](https://commoncrawl.org/terms-of-use) say crawled content may be subject to separate terms, that you must respect the copyrights and other rights of third parties, and that you bear the risks of using crawled content. So the licence question for legal text in Common Crawl is the same as for scraping the source yourself, with one extra difficulty: you no longer know which terms applied, because the crawl does not record them.

For legislation this is often manageable, because statutes are outside copyright in most European countries and EU content is licensed for reuse under the EUR-Lex legal notice. It is not manageable for everything. UK court judgments on Find Case Law are a clear case: the Open Justice Licence does not cover computational analysis, and [The National Archives lists](https://caselaw.nationalarchives.gov.uk/when-you-need-permission) "machine learning or AI training on judgment datasets" as an example needing a separate licence. Copies of those judgments in a web crawl do not escape that. Text and data mining opt-outs and the AI Act copyright policy are covered in [EU copyright, TDM opt-outs and the AI Act for legal data](/guides/ai-act-copyright-tdm-opt-out-legal-data/).

There is also the AI Act. A provider of a general-purpose AI model must describe crawled data in Section 2.3 of its [training-content summary](/guides/ai-act-article-53-training-data-summary-template/), and licensed private datasets in Section 2.2. A curated corpus with a datasheet and per-source licence record is simpler to document than a filtered crawl.

## When is Common Crawl enough, and when is it not?

| Use | Common Crawl / web corpora | Curated legal corpus |
|---|---|---|
| General pre-training | Good: breadth, scale, free | Useful as a high-quality domain mix |
| Continued pre-training on law | Weak: unknown coverage, boilerplate, duplication across crawls | Good: complete, deduplicated, clean text per jurisdiction |
| Supervised fine-tuning | Poor: no structure to build instructions from | Good: articles, metadata and relations to build tasks |
| RAG | Poor: no stable ids, no versions, no citations | Good: article-level chunks with ids and dates ([how to chunk](/guides/legal-rag-chunking-article-level/)) |
| Evals | Poor: no gold sets, and contamination is likely | Good: gold sets from the citation graph ([how to build one](/guides/legal-eval-set-citation-graph/)) |
| Compliance documentation | Hard: no record of terms per page | Simple: source, licence and attribution per record |

A common pattern is to combine the two: web data for breadth in pre-training, and a curated legal corpus for continued pre-training, fine-tuning, retrieval and evaluation. If you do, deduplicate across them on normalised text, or the same acts will be over-weighted.

## Can you build the curated corpus yourself?

Yes. The official sources publish their legislation online and most offer an API or bulk XML; the per-country picture is in [National legislation datasets in Europe](/guides/national-legislation-datasets-europe/). The cost is engineering, not licence fees: discovery, parsing into articles across formats and languages, identifiers, version tracking, relations, withdrawals and monthly upkeep for every source. The trade-off for EU law is set out in [EU legislation datasets for LLM training](/guides/eu-legislation-datasets-llm-training/).

## Key facts

- Common Crawl is a non-profit founded in 2007; its homepage cites over 300 billion pages, with new crawls roughly monthly in WARC, WAT and WET formats.
- FineWeb (15 trillion tokens) is built from 96 Common Crawl snapshots; C4 was introduced with T5.
- CCBot follows robots.txt, and some official legal portals, such as wetten.overheid.nl, block it and other AI crawlers.
- Common Crawl grants no rights in the crawled content; you inherit each source's terms without a record of them.
- Web captures have no article structure, stable legal identifiers, version dates or typed relations, which RAG, fine-tuning and evals need.

## Frequently asked questions

### Does Common Crawl include EUR-Lex?

It may include some EUR-Lex pages, but coverage is not guaranteed and changes from crawl to crawl. Query the Common Crawl index for the domain to see what a given crawl holds. For complete EU coverage, EUR-Lex and the Cellar repository behind it are the primary sources.

### Is legal text in Common Crawl free to use for AI training?

Common Crawl does not license the content; each page keeps its original terms. Most European legislation is outside copyright or openly licensed, but some legal sources, such as UK court judgments on Find Case Law, need a separate licence for AI training wherever you obtained the copy.

### Why not just filter FineWeb for legal domains?

You would get legal text without structure, identifiers, dates or versions, with unknown coverage per jurisdiction and boilerplate to remove. It can help pre-training; it will not give you citable chunks for RAG or gold sets for evals.

## Where Arieta Data fits

Arieta Data is a curated European legal corpus built from official sources through their documented channels. Snapshot 2026-11 holds 419,355 documents from 16 jurisdictions, in 24 languages (EU documents are almost all in English; the national lanes cover 14 national languages): EU legislation, [CJEU](/glossary/cjeu/) case law and national legislation from 15 countries, with approx. 4.07M articles, approx. 2.05B tokens and 801,149 typed citation edges. Every record carries its source, licence and attribution; raw upstream web pages are never shipped. It comes as the [corpus](/datasets/corpus/), the [history](/datasets/history/) and the [graph](/datasets/graph/), with per-source terms on the [coverage page](/coverage/) and the sourcing method on the [provenance page](/provenance/). National case law is included, delivered with personal data masked. The free open sample carries each record's upstream licence and contains no case law; CJEU case law is in the licensed corpus, and a CJEU case-law excerpt is available under an evaluation licence on request.

[Request a sample](/request-access/) and compare it with what your web data holds. We reply within one business day, and a custom cut by jurisdiction, language, date range or document type is ready within 5⁠–⁠6 business days.

*Not legal advice. Check each source's terms and take your own legal advice before training on any dataset.*