İçeriğe geç

A long-context benchmark on Turkish court decisions

Nine language models, more than 27,000 questions. Does a model find a fact as reliably in the middle of an 80-page bundle of Turkish court decisions as at its start or end?

Open data and code on GitHub. Full write-up in Turkish at Prompt, bağlam ve dikkat eksikliği. Measured 13 to 15 November 2026.

Summary

  • Two versions of one task, an easy one and a hard one, built on real decisions of the Turkish Court of Cassation. Nine models, three document lengths, five answer positions, two question registers, three prompt layouts.
  • On the easy version every model scored between 97.5% and 100%. Position and length made no difference up to 80 pages.
  • On the hard version accuracy ranged from 99% for the strongest model to between 10% and 27% for the weakest. Three of nine models lost information in the middle of the document. The four strongest did not.
  • Document length alone did not hurt any model. Crowding did: the shortest documents, which were the densest in near-miss sentences, were the hardest for three models.
  • No model invented an answer for an entity that was absent from the document. The common failure was the opposite one, answering "not in the text" for information that was there.

Why this benchmark

Lawyers upload a long decision, or a folder of decisions, and ask a question. The answer comes back in seconds and reads confidently. Whether the model weighed every page equally is not visible from the answer.

The question has been studied in English. Liu and colleagues (2023) moved the document holding the answer through a list of documents and found that accuracy was highest at the start and the end and dropped in the middle. Chroma's Context Rot report (2025) tested 18 models and found results becoming less reliable as input grew, and a single distractor sentence lowering accuracy. NoLiMa (2025) removed the word overlap between question and target sentence. At 32,000 tokens, 11 of 13 models fell below half of their short-context score.

Those studies used English text and the models of their time. This benchmark asks the same questions of current models on Turkish legal text, where a single word such ashükümsüzlüğünün can take anywhere from three to nine tokens depending on the tokenizer.

What it measures

It measures finding a fact in legal text, not legal reasoning. The model returns a date, an amount, a number of days, a percentage, a file number or a city. It does not decide whether a deadline has passed or whether a party is in default.

Legal knowledge enters in one place, the wording of the question. When the decision saystebellüğ etmiştir ("was served with") and the question asks eline ne zaman geçti ("when did it reach them"), the model has to know these describe the same event.

Three questions were fixed before the main runs.

  1. Is information in the middle of the document missed more often than at the start or the end?
  2. Does accuracy fall as the document gets longer?
  3. Does it matter where the question is placed and how it is worded?

Documents

Documents are built from published decisions of the civil chambers of the Court of Cassation. A pool of 4,697 eligible decisions was drawn with a fixed seed: between 1,200 and 16,000 characters, at least four paragraphs, held-out evaluation sets excluded. Remaining personal names were reduced to initials. Decisions are concatenated under the headings "KARAR 1", "KARAR 2" and so on, and cut at a paragraph boundary at the target length.

LengthTokens (o200k)CharactersWordsDecisionsDocuments
about 5 pages3,98513,0261,6073.240
about 20 pages15,98653,0976,62214.840
about 80 pages63,917212,47926,58661.720

Figures are averages for the hard version. Length is counted with one reference tokenizer for all models, OpenAI's o200k_base. The provider's own reported input token count was recorded on every call and compared with the expected count, to confirm that the whole document reached the model. No call fell below the expected range.

Inserted sentences

Twenty hand-written templates cover procedural and contractual events that occur in ordinary civil cases: service of a payment order, the amount of an amended claim, the date of default, the date a tenant vacated, a penalty clause, a notice period owed to an insurer, the start date of interest, a percentage of fault, security for an interim injunction, waiver of a claim, termination of employment, years of seniority, an on-site inspection, an enforcement file number, an attorney fee, a choice of forum, the end of mediation, the maturity of a promissory note, notice of defect, and an award of damages.

Each document receives five sentences from one template, at the start, the quarter mark, the middle, the three-quarter mark and the end. The five sentences name five invented companies with five different values. The model is asked about one of them. The document does not change between questions, only the company asked about, so the position of the answer is the only thing that varies within a document.

Davalı Zernova Lojistik A.Ş., ödeme emrini 14.03.2019 tarihinde tebellüğ etmiştir.

The defendant Zernova Lojistik A.Ş. was served with the payment order on 14.03.2019.

Company names are coined and do not refer to real companies. No name root occurs anywhere in the decision pool, and no inserted value occurs elsewhere in its document.

Easy and hard versions

In the easy version the target sentence, shown above, carries both the company name and the words of the question. A model can answer by matching the name and the verb.

The hard version changes three things.

  1. No word overlap. The sentence holding the value shares none of the question's distinctive words. The question still asks when the payment order was served. The text reads:İcra müdürlüğünce gönderilen ödeme emri, 12.03.2019 itibarıyla davalı şirkete usulüne uygun olarak ulaşmıştır ("The payment order sent by the enforcement office duly reached the defendant company as of 12.03.2019").
  2. Indirect reference. The company is named once, in an introduction sentence. The value sentence comes three to six paragraphs later in the same decision and refers to the company only by its role, "the defendant company". Where a decision introduces several companies, roles are numbered: "the first defendant", "the second defendant".
  3. Near-miss records. Each target has five distractors of the same shape: two companies with a near-identical name (same root, different sector), two other events for the same company, and one claim that the court did not accept ("…ileri sürmüş ise de bu iddia dosya kapsamıyla doğrulanmamıştır"). Two sit in the target's own decision and three in other decisions. A document holds 5 targets and 25 distractors, 60 inserted sentences in all, every value different.

In the 5-page documents the inserted sentences make up about a third of the text. In the 80-page documents they make up about 2%.

Questions and prompt layout

Every target is asked in two registers.

RegisterQuestion
LegalZernova Lojistik A.Ş. ödeme emrini hangi tarihte tebellüğ etmiştir?
EverydayÖdeme emri Zernova Lojistik’in eline hangi gün geçmiştir?

The instruction is fixed and identical in every call. In translation: "Below are court decisions. Answer the question using only this text. Write only the information asked for (a date, an amount, a number, a file number or a city). If the information is not in the text, write only 'not in the text'." Each call is a single message with no history and no tools. Three layouts were tested: question after the document, question before it, and question in both places.

Grid and controls

Arm5 pages20 pages80 pages
Question after the document400400200
Question before the document200200100
Question in both places200200100

Counts are calls per model in the hard version: documents × 5 positions × 2 registers. With the two controls the hard version comes to 2,240 calls per model and the easy version to 1,360.

  • Absent entity. In 40 documents the model is asked about a sixth company that does not appear in the document. The correct answer is "not in the text".
  • Short baseline. The same questions are asked with the court decisions removed, leaving only the inserted sentences. This shows whether a model can do the task at all, apart from the long context.

Scoring and uncertainty

Scoring is by rule. No model acts as a judge. Values are extracted from the answer and normalised (date formats, month names, thousands separators, currency and percent signs, letter case). Each answer falls into one class.

  • Correct: the value of the company asked about.
  • Confusion: another value from the document, sub-typed as near name, other event, unverified claim, another target, or another target's distractor.
  • Said absent: "not in the text" or an equivalent.
  • Other: a value from nowhere, or a malformed answer.

The first scoring rule counted an answer wrong when it gave the correct value together with a second one. The accuracy table below counts those answers as correct, for every model. The change matters for one row, GLM-5.3 Flash, by 3 to 5 points. For the other models it is under one point.

The unit of analysis is the document. Differences are paired, with a bootstrap over documents of 2,000 resamples and a fixed seed, reported as 95% intervals. A difference is called separated when its interval excludes zero. No correction for multiple comparisons was applied.

Models

ModelIdentifierAccessVersions
GPT-6 Astragpt-6-astrasubscriptionboth
GPT-6.1 Terragpt-6.1-terrasubscriptionboth
GPT-6 Lunagpt-6-lunaAPIboth
DeepSeek V4.1 FlashDeepSeek-V4.1-FlashAPIboth
GLM-5.3 FlashGLM-5.3-FlashAPIboth
Gemini 4 Hydrogengemini-4-hydrogensubscriptionhard
Gemini 4 Carbongemini-4-carbonsubscriptionhard
Claude Sonnet 5.5claude-sonnet-5-5subscriptionhard, no 80 pages
Claude Haiku 5.5claude-haiku-5-5APIhard

Each model ran once, at one setting, with the provider's default reasoning level and temperature 0 where it could be set. Subscription routes prepend a fixed system prefix of a few hundred to a few thousand tokens. The prefix is the same in every call on a route, so comparisons within a model are unaffected. Comparisons of level between models should be read with that difference in mind.

Results, easy version

Accuracy, question after the document.

Model5 pages20 pages80 pages
GPT-6 Astra100100100
GPT-6.1 Terra10099.8100
GPT-6 Luna99.598.899.5
DeepSeek V4.1 Flash10010099.5
GLM-5.3 Flash99.510099.5

Position made no difference and neither did length. Everyday-register questions were answered as well as legal-register ones, within one point. All five models scored 100% on the short baseline and answered "not in the text" 40 times out of 40 on the absent-entity control.

One effect did appear. With the question placed before a 20-page document, GPT-6 Luna fell from 98.3% to 85.8% and DeepSeek V4.1 Flash from 100% to 81.7%. Most of the wrong answers were "not in the text".

Pilot rounds showed that adding distractors alone did not move the strongest models. Raising the number from five to eight left them at 40 out of 40. Removing word overlap and adding indirect reference did.

Results, hard version

Accuracy, question after the document.

Model5 pages20 pages80 pagesShort baseline
GPT-6 Astra99.898.810098.0
GPT-6.1 Terra94.090.093.595.5
Gemini 4 Hydrogen90.290.893.087.5
Gemini 4 Carbon89.590.891.589.0
Claude Sonnet 5.574.087.0not run74.5
GPT-6 Luna76.576.875.564.0
Claude Haiku 5.565.577.275.069.5
GLM-5.3 Flash60.265.066.057.0
DeepSeek V4.1 Flash9.826.522.029.0

The short baseline column is the score on the same questions with the court decisions removed, under the first scoring rule. Five models score below 80% there. For those models the long-document columns measure the difficulty of the task together with the difficulty of long context.

Position

Hard version, 80-page documents. Accuracy by position of the answer.

The table gives the mean of the three middle positions minus the mean of the two ends, in points, with 95% intervals. An asterisk marks a separated difference.

Model5 pages20 pages80 pages
GPT-6 Astra−0.40.00.0
GPT-6.1 Terra−1.7+1.0−4.6 [−12.5, +0.8]
Gemini 4 Hydrogen−2.7 [−5.2, −0.6] *+2.3+0.8
Gemini 4 Carbon+2.3−1.9−1.7
Claude Sonnet 5.5+2.5−0.8not run
GPT-6 Luna−11.9 [−18.3, −5.0] *−10.6 [−18.5, −2.7] *−26.2 [−40.4, −11.7] *
Claude Haiku 5.5−2.7−2.7−5.8 [−12.1, +0.4]
GLM-5.3 Flash−6.9−11.7 [−20.4, −2.9] *−9.6
DeepSeek V4.1 Flash−7.7 [−12.7, −3.1] *−23.5 [−32.9, −14.6] *−27.9 [−39.6, −15.8] *

GPT-6 Luna and DeepSeek V4.1 Flash lose the middle at every length, and the loss grows with length. GLM-5.3 Flash separates at 20 pages. In Luna the effect holds when the analysis is restricted to items the model got right on the short baseline (−10.4 at 20 pages, −19.6 at 80 pages). The comparison is made within each model, the same question at five positions, so a low overall level does not produce it.

Length

No model was worse at 80 pages than at 5. The difference between 80 pages and 5 pages was near zero for Astra, Terra and Luna, and positive for Hydrogen (+2.8), Haiku (+8.8) and DeepSeek (+12.2). Sonnet gained 13 points from 5 to 20 pages.

Part of this is the design. The 5-page documents hold the same 60 inserted sentences as the longer ones, packed into three decisions, with up to twelve numbered "defendant companies" in a single decision. The shortest document is also the most crowded. The benchmark cannot separate the effect of length from the effect of crowding, and nothing longer than 80 pages was tested.

Where the question is placed

Accuracy on 20-page documents, same documents and items in each column.

ModelAfterBeforeBoth
GPT-6 Astra100100100
GPT-6.1 Terra87.596.093.5
Gemini 4 Hydrogen91.095.093.5
Gemini 4 Carbon88.091.590.0
Claude Sonnet 5.586.590.089.5
GPT-6 Luna77.575.081.0
Claude Haiku 5.576.079.577.0
GLM-5.3 Flash63.567.569.0
DeepSeek V4.1 Flash23.514.519.5

No single rule came out. Terra and Hydrogen did better with the question first (separated, by 8.5 and 4.0 points). DeepSeek did better with the question last (9.0 points). The rest did not separate. Putting the question in both places beat putting it last only for Hydrogen, by 2.5 points. The two Claude models showed no separated effect of placement. This table uses the first scoring rule, which affects only the GLM row.

How the question is worded

Accuracy by register at 5, 20 and 80 pages, question after the document.

ModelLegal registerEveryday register
GPT-6 Astra99.5 / 97.5 / 100100 / 100 / 100
GPT-6.1 Terra90.5 / 82.5 / 90.097.5 / 97.5 / 97.0
Gemini 4 Hydrogen81.0 / 83.5 / 86.099.5 / 98.0 / 100
Gemini 4 Carbon85.5 / 88.0 / 89.093.5 / 93.5 / 94.0
Claude Sonnet 5.552.0 / 75.5 / not run96.0 / 98.5 / not run
GPT-6 Luna62.5 / 68.5 / 67.089.5 / 85.0 / 84.0
Claude Haiku 5.538.5 / 61.0 / 56.092.0 / 92.0 / 92.0
GLM-5.3 Flash32.0 / 40.0 / 42.078.5 / 81.0 / 84.0
DeepSeek V4.1 Flash2.0 / 14.0 / 10.017.5 / 39.0 / 34.0

This ran against the prediction. Everyday-register questions were answered better by every model. The most likely explanation lies in the design. In the hard version a legal term such astebellüğ does not appear in the target sentence. It appears only in the near-miss sentences. A question that carries the term pulls the model toward a distractor, or leads it to answer "not in the text" because the term is not used for that company. An everyday-register question carries no such term. This explanation was not tested with a separate arm.

Read this way, the column measures what happens when the words of the question occur verbatim somewhere other than the right sentence. The ranking of the two Claude models in the accuracy table largely reflects this effect. On everyday-register questions Sonnet scores 96% to 98.5% and Haiku 92%.

Kinds of error

  • No fabrication on the control. Asked about a company that was not in the document, every model answered "not in the text" 40 times out of 40, in both versions.
  • False "not in the text". For the stronger models nearly all errors are of this kind. Of GPT-6.1 Terra's 77 wrong answers with the question after the document, 76 were "not in the text" for information that was present. Both Gemini models made no confusion errors in 2,000 long-document calls. Sonnet made one.
  • Taking a claim for a fact. Weaker models also went to the wrong sentence. Of GPT-6 Luna's 164 confusions, 137 gave the value from the sentence reporting a claim the court had not accepted. For Claude Haiku 5.5 the share was 88 of 105.
  • Near names. DeepSeek V4.1 Flash confused companies with near-identical names more often than any other kind of distractor, 131 of 376 confusions.

Predictions and what held

The plan and the expected directions were committed before the first call of each main run.

PredictionOutcome
The middle is missed more oftenHeld in three models on the hard version. Not on the easy version.
Longer documents lower accuracyDid not hold in either version.
Question last is at least as good as question firstHeld on the easy version. Mixed on the hard version.
Everyday wording lowers accuracyNo difference on the easy version. Reversed on the hard version.

What to keep in mind

  • Each model was run once, at one setting. There is no estimate of run-to-run variation.
  • The inserted sentences are synthetic. The surrounding text is real.
  • The four models added later were run only on the hard version. Sonnet was not run at 80 pages.
  • Claude Opus 5.5 is still running and is not in the tables.
  • The accuracy table uses the revised scoring rule. The position chart, the position table and the placement table use the first rule. The two differ only for GLM-5.3 Flash, by at most 3 points.
  • Paid API cost was small: 2.60 US dollars for the easy version, 7.27 for the first five models on the hard version, and 9.06 for Claude Haiku 5.5. The remaining models ran on subscription routes.

Data, code and reuse

The public repository holds one folder per version, each with the same files.

FileContents
nasil-olculdu.mdMethod, in Turkish
ozet.jsonEvery table, with intervals
sonuclar.jsonlOne row per call: model, arm, length, document, template, position, register, layout, class, reported input tokens
sablonlar.jsonTemplates for the inserted sentences and the questions
kos.pyScript that builds a document, asks the question and scores the answer

The court decisions and the raw model answers are not published. The script takes paths as arguments, so the benchmark can be rebuilt on any collection of Turkish decisions, including the publicTurkish court decisions corpuson Hugging Face. Files in the repository were produced with the first scoring rule.

Where this is going

This benchmark is one piece of a larger aim, an open benchmark for Turkish legal AI that anyone can run. A researcher, a small legal-tech team or a law faculty should be able to test three things: whether a model finds the strongest ruling against an argument, whether it cites the court's actual reasoning, and whether it recognises when different facts make a ruling distinguishable.

The first public release is planned to contain three parts.

  • A runnable evaluation harness.
  • A documented scoring rubric.
  • A small set of lawyer-reviewed examples built on public decisions. Each example links to the official ruling and explains why it contradicts the claim or why it does not.

Teams would run the same tests on their own models without sending anyone their briefs. Client material, the private corpus and the held-out test cases stay private. The purpose is a shared test that makes it harder for Turkish legal AI to look good by telling lawyers what they want to hear.

Planned automation with Devin

Nothing on this page was produced with Devin. What follows is the plan for the next stage, which is to run the benchmark continuously with Devin bots.

  • Weekly adverse-ruling watch. Every Monday a bot pulls public rulings from selected chambers of the Court of Cassation and uses official decision identifiers to find what is new, including older rulings published late. It searches them against a watchlist of anonymised legal arguments. For each argument it flags, it reports the strongest adverse ruling, the official link, the exact paragraph of the court's reasoning, and a short note on whether the court rejected the same argument or reached a different result because the facts or the applicable law differed.
  • Temporal validity checks. A test suite checks whether a system cites the version of a provision that was in force on the relevant date.
  • Reproduction on regression. When a benchmark score drops, the bot opens a pull request with a reproducible test and a proposed fix.

Related work on this site

The benchmark is part of a series in Turkish on how language models read, retrieve and write Turkish legal text. Related pieces coversix tokenizers on Turkish legal text,five BERT-family encoders on a statute classification task, andrule-based against trained citation extraction. A fine-tuned citation extraction model is published asgliner-tr-legal-extraction.

References