Omissions and form, not arithmetic

The verdict is strict: one failed rubric item in a critical requirement fails the task; 2 of Opus 5.5's 3 failed tasks fail only on an exact label.

* The rubric is stricter than the expert reference here.

† Depends on the additions rule (ours, set before the runs: additions listed line by line); without it, tasks: Opus 5.5 7 pass / 2 fail / 1 undecided, GPT-6.1-sol 3 / 6 / 1.

Weeks of supply fails for both models only on the chart: the pre-registered chart requirement asks for exactly the five UPCs as categories; the inventory measures (scope, rate and weeks of supply, out-of-stock) pass for both.

Borderline cases. For 12 of the 40 failures, whether the prompt asked for the missing piece is a judgment call (the author decided each). Counting all 12 as rubric only, GPT-6.1-sol fails 1 task on something the prompt asked for; counting all as in the prompt, 4. All 12 are GPT-6.1-sol failures.

How the 35 critical requirements were chosen. They were set in the preregistration before the evaluation runs, 3 to 5 per task. Each names a must-meet part of the task whose failure matters most to the business outcome and cannot be offset by other points, recorded with its evidence location and test method.

• Before review: the graders' verdicts plus the pre-registered human visual review.
• After review: plus author review, which corrected grader errors and settled every critical requirement the graders left undecided. This is the headline.

Review. Every item was confirmed by the author's review. Category C failures were all flagged first by the LLM judge or the blind audit, none by a code grader.

Tour profit and loss. GPT-6.1-sol's two missing per-stop rubric items follow the expert reference's country-level approach; the rubric is stricter than the reference.

Weeks of supply. GPT-6.1-sol's six missing daily figures are the input to the weekly rate, which it reports correctly; that critical requirement passes. The failure type's high severity applies to the store subtotals and per-stop amounts.

Single occurrences. A failure type seen once shows that the failure happened, not how often.

How it targets each gap

Three uses

• Evaluate. Score completeness against the checklist separately from numerical accuracy, and items in the prompt separately from rubric-only ones.
• Train. Expert deliverables as demonstrations; complete versus deficient pairs for preference training; the automated tests as rewards for reinforcement learning.
• Guard. Rerun as a regression suite on each new model version.

Observed failure

GPT-6.1-sol, weekly sales analysis, rubric items 16 / 25 / 26: 'Sales by Store' holds every store-brand row and a grand total that reconciles to the source, but no store subtotal rows, although the prompt asks the tab to "total the sales by store"; the store-level view cannot be read without re-aggregating. Claude Opus 5.5's workbook carries a labelled total for each of the 39 stores.

Task input

a weekly sales export (936 rows: store, brand, WTD / MTD / YTD sales quantity, sales dollars and stock on hand) and the instruction to build brand and store summaries with sell-through as pivot tables.

Business rule

a store-level view carries one labelled subtotal per store equal to the sum of its brand rows, and a grand total equal to the sum of the subtotals; sell-through is the ratio of sums at each level.

Checklist (excerpt)

Sales by Store is a pivot table built from the Data tab (in the prompt); a labelled total for each store and a grand total equal to their sum (in the prompt: "total the sales by store", "include grand totals"; rubric items 16 / 25 / 26); ST% = sales ÷ stock on hand at every level (in the prompt), computed from the totals, not averaged over rows (an evaluator's rule).

Target deliverable

on the Sales by Store tab, a pivot table built from the Data tab with Store and Brand as row fields, a labelled total under each store block and a grand total, each reconciling to the export.

Automated test

Sales by Store holds a pivot table whose source is the Data tab; for each of the 39 stores a labelled total equal to the independent recomputation from the export; the grand total equal to the sum of the store totals; ST% within 0.0005 of the ratio of sums; expected values computed from the source, not read from the deliverable.

Why verified

GDPval's own expert file for this task fails two of these tests: its ST% averages row ratios (Clinique WTD 0.2467 against 0.2309 from the totals), and its Data tab differs from the source export in 557 cells, a difference the audit could not explain (an expert edit or another input version).

Targeted failure type

dropping a requested aggregation level during presentation while the top-level totals stay correct.

Composition

the evaluation’s 10 tasks and three occupations (Order Clerks; Accountants and Auditors; Computer and Information Systems Managers) as the seed, extended with tasks of the same families built from real source data, each with an expert deliverable and its checklist.

Training use

complete / deficient pairs, the same task with and without the required level (the expert deliverable against a version that drops the level), labelled by the checklist.

Train / evaluation separation

within each family, training and held-out tasks use different source materials (no shared sales export, store list or rubric text); held-out tasks use the same frozen tests. A separate leave-one-family-out run measures transfer to a new family.

Hypothesis

Expert deliverables carrying every requested level, with the task’s checklist, raise level-presence pass rates on unseen same-family tasks without lowering top-line accuracy; untested.

1. 1

   Rubric. Each GDPval task comes with an expert rubric: a list of rubric items, each worth points, such as one requiring a subtotal row for each store. Each model faces 580 rubric items across the 10 tasks. Before any model ran, the author grouped the rubric items each task can't do without into critical requirements: 35 in all, 3 to 5 per task, covering 320 rubric items.

   Example: the weekly sales task

   4 critical requirements, with their rubric items paraphrased: the right sheets (such as a sheet named Sales by Store); correct brand totals (one row per brand; each value equals the sum of the source rows); a store view with a subtotal per store (a subtotal row for each store; the grand total equals the sum of the subtotals); and correct sell-through (WTD ST% = WTD sales quantity ÷ WTD stock on hand).

2. 2

   Run. Each model did each task once, with the same prompt, tools and offline sandbox, at high reasoning effort (the same label doesn't mean the same compute), up to 32,000 output tokens per request and a two-hour limit.

3. 3

   Grade. Each rubric item went to one grader: a code grader, an LLM judge or human visual review. Graders never guess: if one couldn't read the part of the file it needed, the verdict is cannot assess, not a fail (e.g. content inside a chart, or two columns that both look like the vendor).

4. 4

   Review. An AI draft (OpenAI Codex) first sorted the 239 rubric items graded fail or cannot assess by likely cause. In author review, the author then reviewed 170 rubric items by hand: every failed or cannot-assess rubric item that could change a critical requirement's verdict, every likely model failure, and the one graded pass the blind audit found failing.

1. Prepare. Selected the 10 tasks, checked their source files, rubrics and expert references, and fixed the 35 critical requirements in a preregistration.
2. Calibrate. Tuned the code graders on development runs, then tested them on held-out runs before freezing.
3. Freeze. Fixed the prompt, graders and configuration (tags freeze-v1-generation and freeze-v1-grading).
4. Run. anthropic/claude-opus-5-5 (adaptive thinking, effort high) and openai/gpt-6.1-sol (reasoning effort high), one attempt per task, 32,000 output tokens per request, two-hour limit, up to 3 retries on transport errors only; 2 Nov 2026, 09:57–11:04 UTC.
5. Grade. Code graders recompute expected values from the source files; the LLM judge (Gemini 3.1 Pro) grades the rest from the relevant cells, pages and source values; human visual review covers charts and layout. Judging ran 11:57–12:55 UTC.
6. Audit. The author labelled 20 rubric items blind (saved 12:48 UTC, before seeing results). The LLM judge's held-out validation (3/3) ran at the same time as the main judging.
7. Review. AI draft (OpenAI Codex, read-only) over the 239 rubric items graded fail or cannot assess; author review of 170, about 3 hours, with cells and pages located by Claude.

• Graders never guess. "Cannot assess" instead of an inferred pass, so no hidden passes.
• The expert reference is evidence too. Expected values come from the source files, which exposed a rubric stricter than its reference (tour task).
• Frozen verdicts, author decisions on top. Every correction stays traceable, and review time goes only where it can change a verdict.
• Task pass as the headline. A deliverable counts only if every must-have holds; severity shows how bad each failure is. Trade-off: one exact-label miss fails a whole task.

• Which 170. Of the 239 rubric items graded fail or cannot assess, the 145 inside critical requirements and the 24 the AI draft suspected were model failures; plus 1 graded pass that the blind audit found failing. By graded verdict: 100 fail (67 grader false fails, 33 confirmed), 69 cannot assess (21 pass, 6 fail, 42 left) and the 1 pass.
• Not reviewed: 96. 60 graded fail outside critical requirements (the AI draft: 29 grader false fails, 31 ambiguous rubric items), 10 cannot assess outside critical requirements, and 24 visual items without a verdict plus 2 invalid judge outputs.
• AI draft. OpenAI Codex (read-only) sorted the 239 rubric items graded fail or cannot assess by likely cause. The author decided each of the 170 from the deliverable itself, or from cells and pages located by Claude, with the AI draft only as a second opinion.
• Grader false fail, example. The code grader looked for "on hand" and missed "On-Hand", so ten correct inventory totals were judged missing.