How to chunk legal documents for RAG: by article, not by token count

For legislation, chunk by the document’s own structure: one chunk per article, or per numbered paragraph when an article is long, with the act’s identifier, title, article number and version date attached as metadata. Fixed token windows are the default in most RAG frameworks, but on legal text they cut rules away from their exceptions, merge unrelated provisions and lose the citation a grounded answer has to give. An article, by contrast, is the unit lawyers cite and courts interpret, so an article-level chunk can be quoted, checked and linked back to its source. Arieta Data’s 2026-11 snapshot ships approx. 4.07M articles from 419,355 EU and national documents, already split this way.

Why does chunking matter more for legal RAG?

Retrieval-augmented generation, introduced by Lewis et al. (NeurIPS 2020, arXiv:2005.11401), conditions a generator on passages retrieved from an index. The answer can only be as good as the passages, and in law a passage is only useful if it is complete and citable.

Legal research tools built on RAG still make mistakes. Magesh et al. (arXiv:2405.20362) tested legal research assistants from LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) and found that they each hallucinated between 17 and 33 percent of the time; the study was published in the Journal of Empirical Legal Studies in 2025. Chunking is not the only cause, but it decides whether the retriever can return the provision that answers the question, and whether the generator can cite it.

Three properties of legal text make naive chunking fail:

1. Rules and exceptions sit together. Article 17 of the GDPR sets out the right to erasure in paragraph 1 and its exceptions in paragraph 3. A 512-token window can return paragraph 1 without paragraph 3, and a model will then state the right without its limits.
2. Cross-references are everywhere. “Without prejudice to Article 6(1)” is meaningless without Article 6. A chunk needs its identifier so the referenced provision can be fetched.
3. Text changes over time. The same article number can carry different text in different versions. A chunk without a version date cannot say what the law was on a given day.

What is wrong with fixed-size token chunks?

Fixed-size chunking with overlap is simple and language-agnostic, which is why it is the default. On legislation it has predictable failure modes:

Failure	Example	Effect on answers
Rule split from exception	Paragraph 1 in one chunk, its derogations in the next	Confident, incomplete answers
Two provisions merged	The end of Article 5 and the start of Article 6 in one chunk	Wrong article cited; rules blended
Lost heading	The article title or chapter falls in the previous chunk	Weaker retrieval for topical queries
Lost identity	The chunk has text but no act or article number	No citation, so no way to verify
Overlap duplicates	The same sentence in two chunks	Redundant context; skewed reranking

Long-context models do not remove the problem. Liu et al., “Lost in the Middle” (TACL 2024, arXiv:2307.03172), found that models use information at the start and end of a long context better than information in the middle. Sending whole acts to the model wastes context and buries the provision that matters.

What should a legal chunk contain?

A chunk should be the smallest unit that a lawyer would cite and that still makes sense on its own. For EU and most national legislation that is the article. EU acts follow a fixed drafting structure: guideline 15 of the Joint Practical Guide for drafters of EU legislation says the enacting terms are subdivided into articles (and, for long acts, titles, chapters and sections), and the Guide subdivides articles further into paragraphs, subparagraphs and points, and national legislation uses equivalent units: sections in UK Acts, Paragraphen in German statutes, articles in French codes.

Field	Why it is there	Arieta Data field
Document id	Join with the source and other datasets; resolve cross-references	id / celex (a CELEX number for EU documents)
Article id and number	The citation: “Article 17(3)”	articles[].id, articles[].number
Article heading	Strong signal for topical queries	articles[].title
Article text	The content, with no HTML	articles[].text
Act title and type	Context for the embedding and for the answer	title, documentType
Jurisdiction and language	Filters; multilingual retrieval	jurisdiction, language
Dates	Point-in-time filtering	dateOfDocument, dateOfEffect, dateOfPublication
Version and hash	Reproducibility; detecting changed text	version, contentHash
Source and licence	Attribution in the answer and the model card	source.*

Recitals deserve their own chunks. They are not binding, but they explain the purpose of the provisions and courts use them to interpret articles. Index them separately and label them, so a system can use them for context without presenting them as the rule.

How do you handle long articles, annexes and tables?

Most articles fit comfortably in an embedding model’s input. Arieta Data’s 2026-11 snapshot averages out at roughly 500 tokens per article (approx. 2.05B tokens over approx. 4.07M articles, both estimated from a sample), but that is an average, and definitions articles and annexes can be far longer.

A practical rule set:

1. Default to one chunk per article.
2. Split long articles at paragraph boundaries, never inside a paragraph, and repeat the article number and heading in each part (“Article 4(1)–(10), Definitions”).
3. Prefix each chunk with a short header built from metadata: act title, article number and heading. This gives the embedding the context a bare paragraph lacks. Anthropic’s contextual retrieval write-up (September 2024) reports that prepending generated context to chunks cut failed retrievals by 49 percent, and by 67 percent with reranking; for legislation, most of that context already exists as metadata.
4. Treat annexes as documents of their own, split by their own headings or table rows.
5. Keep the parent link. Store the document id with every chunk so you can expand to neighbouring articles, or to the whole act, at answer time.

How do you keep chunks current and point-in-time?

Legislation changes, and a RAG index that holds one version per article will answer questions about the past with today’s text. Store the version and its date on every chunk, and keep earlier versions in the index or alongside it. At query time, filter by date when the question has one (“under the rules in force in 2023”) and otherwise prefer the latest version.

Repealed acts are a related trap. A citation graph with typed repeal and amendment edges lets you demote or filter repealed and superseded text instead of serving it as current. The method is in Point-in-time legal data, and the graph signals in Citation graphs as supervision signal for legal retrieval.

How do you get citations into the answer?

Grounding means the system can show where each statement comes from. Article-level chunks make this mechanical: each retrieved chunk carries a citable reference (act, article, paragraph) and a stable id that resolves to the source. Ask the model to cite chunk ids, then render them as “Article 17(3)(b), Regulation (EU) 2016/679” with a link built from the id. Verify after generation that every cited id was in the retrieved set.

Two practices help:

• Cite the identifier, not only the title. Titles are long and ambiguous across versions; a CELEX number, ECLI or ELI is not. See CELEX, ECLI and ELI explained.
• Expand cross-references at retrieval time. When a retrieved article says “referred to in Article 6(1)”, fetch Article 6 as well. Reference edges from the graph make this a lookup rather than a regular expression.

How should you evaluate legal retrieval?

Measure retrieval separately from generation. LegalBench-RAG (Pipitone and Alami, arXiv:2408.10343) is built around retrieving the precise text snippets that answer a legal question rather than whole documents, which is the right target. For EU and national law, gold sets can come from the citation graph: the articles an act cites, the national measures notified for a directive, or pairs of versions before and after an amendment. Building a legal eval set from a citation graph sets out the method.

Key facts

• RAG was introduced by Lewis et al. in 2020 (arXiv:2005.11401); the unit you retrieve decides what the model can ground on.
• Leading legal research tools each hallucinated between 17 and 33 percent of the time in Magesh et al. (arXiv:2405.20362).
• An article is the unit lawyers cite; one chunk per article, split at paragraphs when long, keeps rules with their exceptions.
• Every chunk needs a stable document id, article number, version date and source, or the answer cannot be verified.
• Arieta Data snapshot 2026-11: 419,355 documents, approx. 4.07M articles, approx. 2.05B tokens, from 16 jurisdictions, in 24 languages (EU documents are almost all in English; the national lanes cover 14 national languages).

Frequently asked questions

What chunk size should I use for legal RAG?

Use the article as the unit rather than a fixed size. Most articles fit an embedding model’s input; split only long ones, at paragraph boundaries, and repeat the article number and heading in each part.

Should recitals be indexed?

Yes, as separate, labelled chunks. Recitals explain the purpose of EU provisions and are used in interpretation, but they are not binding, so a system should not present them as the rule.

Does a long-context model remove the need for chunking?

No. Models use information in the middle of long contexts less reliably, whole acts waste context, and you still need provision-level citations for grounding. Retrieve articles and expand to neighbours when needed.

How do I stop a RAG system citing repealed law?

Store version dates on chunks and join them with typed repeal and amendment edges. Filter or demote repealed acts and superseded versions unless the question asks about the past.

Where Arieta Data fits

Arieta Data’s corpus is EU legislation, CJEU case law and national legislation from 15 countries at article level, with recitals, metadata, dates, content hashes and per-record source and licence. The history adds every observed version for point-in-time retrieval, and the graph adds 801,149 typed citation edges for cross-reference expansion and repeal filtering. Formats are Apache Parquet and gzip JSON Lines; per-lane counts are on the coverage page. National case law is included, delivered with personal data masked.

Request a sample and load it straight into your index. We reply within one business day, and a custom cut by jurisdiction, language, date range or document type is ready within 5⁠–⁠6 business days.

Not legal advice. A RAG system’s answers about law should be checked against the authentic text.