How to build a legal eval set from a citation graph

The cheapest reliable way to build a legal evaluation set is to take the gold answers from relations the law already states, then have lawyers review a sample rather than label everything. A citation graph records which provisions an act cites, which acts amend or repeal it, which implementing acts complete it and which national measures were notified to transpose it. Each of those relations is a question with a known answer. A version history adds dates, so every item can be pinned to what the law said on a given day and checked against changes the model has not seen. Arieta Data’s 2026-11 graph has 801,149 typed edges across EU and national law, including 66,670 transposition edges.

Why are existing legal benchmarks not enough?

Public legal benchmarks are useful, but few of them test what a European legal product does.

Benchmark	What it tests	Coverage
LegalBench (Guha et al., arXiv:2308.11462)	162 legal reasoning tasks	Mostly US law
LexGLUE (Chalkidis et al., ACL 2022, arXiv:2110.00976)	Seven classification tasks, including EU legislation labelled with EuroVoc concepts	English
LEXTREME (Niklaus et al., arXiv:2301.13126)	Multilingual legal classification and named-entity tasks	24 languages
COLIEE	Case-law retrieval and entailment; statute retrieval on the Japanese Civil Code	Canada, Japan
CLERC (Hou et al., Findings of NAACL 2025)	Retrieving the cases a passage of legal analysis cites	US federal case law

Three gaps follow. Few benchmarks cover EU and national legislation as retrieval tasks; most are frozen, so their answers go stale as the law changes; and public test sets leak into training data. Sainz et al. (Findings of EMNLP 2023, arXiv:2310.18018) argue that contamination must be measured per benchmark, and Golchin and Surdeanu (ICLR 2024, arXiv:2308.08493) show that models can reproduce test instances they have seen. A private, dated eval set built from your own jurisdictions avoids all three.

Which questions can a citation graph answer?

Each edge type in a legal citation graph supports a different task with a gold set that no annotator had to write. These are the edge types in Arieta Data’s graph, with their counts in snapshot 2026-11:

Edge type	Edges (2026-11)	Eval task	Gold set
reference	651,075	Citation retrieval: given a passage, find the provision it relies on	The cited document or article
transposition	66,670	Cross-jurisdiction retrieval: given a directive, find the national measures	Measures notified for the directive
amendment	37,922	Change tracking: what amended this act?	The amending acts
legal-basis	33,900	Find the legal basis of an act	The treaty article or parent act
implementation	9,364	Find the detailed rules under a framework act	Its delegated and implementing acts
corrigendum	10	Find corrections to a language version	The corrigenda
repeal	1,444	Is this act still in force?	The repealing act
proposal	764	Link legislative history to the final act	The Commission proposal

The reference edges with a captured citation context give the classic citation-recommendation setup used in LePaRD (Mahari et al., ACL 2024) and CLERC: the text around a citation is the query, and the cited provision is the answer. The structural edges give questions a legal product actually gets: “what implements this regulation?”, “has this been repealed?”, “how did France transpose this directive?”.

How do you turn edges into eval items?

A workable pipeline has six steps.

1. Choose the task and the population. Decide the jurisdictions, languages, document types and date range, and which edge types feed which task. Sample by document as well as by edge, or heavily cited acts will dominate.
2. Write the query. For context-based tasks, use the citation context with the citation itself masked (“…in accordance with [MASK]…”). For structural tasks, use a template (“Which national measures did Spain notify for Directive (EU) 2019/790?”) and paraphrase a sample with an LLM, then check the paraphrases.
3. Collect the gold set. Take all targets of the chosen edge type for that source, not just one, so recall can be measured.
4. Add hard negatives. Use acts with the same legal basis that are not cited, earlier acts the target repealed, and the pre-amendment version of the target article. Drop any candidate connected to the query by another edge, so a true answer is not scored as wrong.
5. Pin a date. Give each item an asOf date and restrict both the retrieval corpus and the gold set to what existed then, using the version history. This is the method in Point-in-time legal data.
6. Review a sample with lawyers. Graph-derived gold sets are precise but not complete, and a citation is not always relevance. Have a lawyer check a random sample of items per task and record the error rate.

How do you score it?

Score retrieval and generation separately, so you know which one failed.

Layer	Metric	Notes
Retrieval	Recall@k, nDCG@10, MRR	nDCG@10 is the headline metric in BEIR (Thakur et al., arXiv:2104.08663); recall@k matters most when the generator sees the top k
Citation	Exact match on the cited id, at document and article level	Identifiers make this mechanical; see CELEX, ECLI and ELI explained
Answer	Correctness against the gold provision, judged by a rubric	Use an LLM judge only after calibrating it against lawyer judgements
Temporal	Accuracy on paired items before and after a change	A system that ignores dates fails one item of each pair

Report results per task, per jurisdiction and per language. An average over a mixed set hides the cases that matter, such as a system that is strong on EU regulations and weak on national transposition.

How do you keep the eval set from going stale or leaking?

Law changes every month, so an eval set needs maintenance like any other dataset.

• Hold out by time. Train or tune on an earlier snapshot and test on items derived from edges and versions that first appear in a later one. That measures how a system handles law it cannot have memorised.
• Keep it private. Do not publish the test set. If you publish results, publish the method and a small sample.
• Re-check on every update. Monthly delta snapshots list what changed; re-derive the items whose gold documents changed or were withdrawn.
• Version the eval set. Record the snapshot id each item was built from, so a score is always tied to a known state of the law.

What are the pitfalls?

1. Absent edge is not a negative. No graph is complete. Treat unlinked candidates as unknown unless you have checked them.
2. Notification is not correctness. A transposition edge records that a member state notified a measure for a directive, not that it transposed the directive completely or correctly. Write questions accordingly; see EU directive transposition as structured data.
3. Boilerplate citations. References to treaty articles in every preamble are weak relevance signals. Filter or down-weight them.
4. Template leakage. If every query follows one template, a model can learn the template. Paraphrase and mix query styles.
5. Personal data. If your eval set uses court decisions, it inherits their personal data. Keep that in your data-protection assessment.

Key facts

• A citation graph turns stated legal relations into gold sets: cited provisions, amending, repealing and implementing acts, and notified transposition measures.
• Arieta Data snapshot 2026-11: 801,149 typed edges, of which 651,075 are references, 66,670 transposition, 37,922 amendment and 33,900 legal-basis edges.
• Public legal benchmarks such as LegalBench (162 tasks) and LexGLUE are mostly US-centred or classification-based, and frozen in time.
• Pin every item to an as-of date and hold out a later snapshot to measure performance on unseen law.
• Graph-derived gold sets are precise but incomplete: review a sample with lawyers and never treat a missing edge as a negative.

Frequently asked questions

Do I still need lawyers to build a legal eval set?

Yes, but far less of their time. The graph supplies the gold answers; lawyers review a sample per task, write rubrics for answer quality and check edge cases, instead of labelling every item from scratch.

How many items does a legal eval set need?

Enough per task, jurisdiction and language to separate the systems you are comparing. A few hundred items per slice is a common starting point; report confidence intervals rather than a single average.

Can I use a public benchmark instead?

Use public benchmarks for comparison with published results, but expect contamination and stale answers. For product decisions, a private, dated eval set on your own jurisdictions is more reliable.

What is a point-in-time eval item?

An item with an as-of date, where the retrieval corpus and the gold answer are both restricted to the versions of documents that existed on that date. It stops later amendments from leaking into earlier questions.

Where Arieta Data fits

Arieta Data’s citation graph and version history are cut from the same monthly snapshot as the article-level corpus, so edges, versions and text join on the same ids; per-lane edge counts are on the coverage page. Each edge carries its type, its origin and, where captured, its citation context, with no probabilistic edges. Arieta Data eval sets are built to your spec on request from the graph and the history: your jurisdictions, languages and dates, with point-in-time answers and held-out snapshots, scoped with you and timed in the quote. National case law is included, delivered with personal data masked.

Request a sample of the graph and history to see the raw material. We reply within one business day, and a custom cut of the graph, history or corpus by jurisdiction, language, date range or document type is ready within 5⁠–⁠6 business days.

Not legal advice. A citation or transposition edge records a relation stated by a source; it is not a legal assessment.