LongListBench Leaderboard

Deep Extract v3 (semantic tune)

Reducto · targeted fields · 2026-07-25 · n=1

Raw PDF · prompt eng

$10.08beta

96.0%

16/32

85.9%

100.0%

98.8%

Deep Extract v3 (semantic tune)32 documents · n=1

Document	Gold records	Predicted	Exact recall	Field F1	Complete
driver_mvr_packet_001	260	260	100.0%	100.0%	Complete
driver_mvr_packet_002	500	500	100.0%	100.0%	Complete
driver_mvr_packet_003	500	500	100.0%	100.0%	Complete
driver_schedule_sparse_001	500	500	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_001	1159	1159	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_002	2143	2143	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_003	2195	2195	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_004	2239	2239	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_005	2379	2379	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_006	2434	2434	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_007	2445	2445	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_008	2571	2571	100.0%	100.0%	Complete
ifta_multisection_return_001	335	335	96.7%	99.9%	Incomplete
ifta_multisection_return_002	461	475	92.0%	98.7%	Incomplete
ifta_return_schedule_001	582	558	95.9%	98.4%	Incomplete
ifta_return_schedule_002	998	962	96.4%	98.6%	Incomplete
ifta_return_schedule_005	1157	1115	96.4%	98.6%	Incomplete
ifta_tax_inquiry_001	650	649	99.8%	99.9%	Incomplete
ifta_tax_inquiry_002	650	649	99.8%	99.9%	Incomplete
ifta_tax_summary_001	760	760	100.0%	100.0%	Complete
ifta_tax_summary_002	760	760	100.0%	100.0%	Complete
loss_run_external_001	300	300	96.0%	99.7%	Incomplete
loss_run_external_002	300	300	91.7%	99.4%	Incomplete
loss_run_external_003	300	300	92.3%	99.7%	Incomplete
mixed_040_001_crosspage	40	40	0.0%	91.5%	Incomplete
mixed_cgl_040_001	562	581	24.0%	81.6%	Incomplete
multihop_012_001_crosspage	12	12	0.0%	91.7%	Incomplete
multihop_025_001_crosspage	25	25	0.0%	91.5%	Incomplete
multihop_bop_012_001	344	341	53.2%	95.7%	Incomplete
multihop_wc_025_001	438	456	28.5%	89.9%	Incomplete
vehicle_schedule_sparse_001	800	800	100.0%	100.0%	Complete
vehicle_schedule_sparse_002	800	800	100.0%	100.0%	Complete

Prompt templatePrompt engineered after observing benchmark failure modes.

Generated field contract

Output JSON shape:
{ "records": [ ... ] }

General requirements:
- Extract every target record visible in the document.
- Use the exact field names below; do not invent fields that are not listed.
{{RECORD_TYPE_REQUIREMENT}}
- If a listed field is not visible for a record, use an empty string.
- Preserve identifiers, dates, codes, locations, limits, rates, totals, names, and premiums exactly as shown.
- Do not deduplicate repeated records unless they are exact duplicate rows for the same item.

Allowed record groups and fields:
{{ALLOWED_RECORD_GROUPS_AND_FIELDS}}

All allowed fields:
{{ALL_ALLOWED_FIELDS}}

Target-scope rules:
- Extract records that match the listed fields, not every nearby note, notice, subtotal, or support table.
- Keep each field to its own value; do not include neighboring labels or carried-over header text.
- For tables split across pages or sections, preserve the inherited row context needed to complete each record.
- For policy packets, extract scheduled locations, classifications, coverages, forms, endorsements, material clauses, rating rows, and premium rows that match the allowed fields.
- For operations documents, extract one record per target table row or section-row combination matching the allowed fields.
{{CONDITIONAL_SCOPE_RULES}}

Prompt engineering additions

Field granularity and fidelity:
- Output only the atomic value each field names. Exclude surrounding labels, caption words that merely restate the field, header/breadcrumb prefixes, page-wrapper text, and annotations.
- Put each distinct attribute in its own field. Do not concatenate a second attribute (a category, unit, fuel/coverage qualifier, etc.) into another field's value.
- When the document shows both a full path/breadcrumb and a specific leaf label for a field, output the leaf label.
- When both a short canonical label and a longer restated narrative refer to the same field, output the short canonical value.
- Only output a value that is actually present for that record; if a listed field is not present, use an empty string. Do not infer, carry over, or fabricate.

Identifier and label fields:
- For fields that name an identifier or label (e.g. a schedule, return, form, or section
  name), output only that identifier itself. Do not append a separate attribute — a fuel
  type, coverage type, product category, or other section qualifier — onto it. Such
  qualifiers are distinct attributes; put each in its own field when the schema provides
  one, and otherwise omit it from the identifier value.

Deep Extract v3 (strict contract)

Reducto · identical contract · 2026-07-24 · n=1

Raw PDF · strict contract

$11.02beta

89.0%

16/32

61.1%

100.0%

98.0%

Deep Extract v3 (strict contract)32 documents · n=1

Document	Gold records	Predicted	Exact recall	Field F1	Complete
driver_mvr_packet_001	260	260	100.0%	100.0%	Complete
driver_mvr_packet_002	500	500	100.0%	100.0%	Complete
driver_mvr_packet_003	500	500	100.0%	100.0%	Complete
driver_schedule_sparse_001	500	500	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_001	1159	1159	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_002	2143	2143	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_003	2195	2195	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_004	2239	2239	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_005	2379	2379	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_006	2434	2434	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_007	2445	2445	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_008	2571	2571	100.0%	100.0%	Complete
ifta_multisection_return_001	335	335	97.3%	99.9%	Incomplete
ifta_multisection_return_002	461	461	96.5%	99.9%	Incomplete
ifta_return_schedule_001	582	558	95.9%	98.4%	Incomplete
ifta_return_schedule_002	998	962	0.0%	90.4%	Incomplete
ifta_return_schedule_005	1157	1115	0.0%	90.4%	Incomplete
ifta_tax_inquiry_001	650	649	99.8%	99.9%	Incomplete
ifta_tax_inquiry_002	650	665	100.0%	98.9%	Incomplete
ifta_tax_summary_001	760	760	100.0%	100.0%	Complete
ifta_tax_summary_002	760	760	100.0%	100.0%	Complete
loss_run_external_001	300	300	80.7%	99.3%	Incomplete
loss_run_external_002	300	300	97.7%	99.8%	Incomplete
loss_run_external_003	300	300	99.7%	100.0%	Incomplete
mixed_040_001_crosspage	40	40	0.0%	91.5%	Incomplete
mixed_cgl_040_001	562	570	27.2%	86.0%	Incomplete
multihop_012_001_crosspage	12	12	0.0%	87.5%	Incomplete
multihop_025_001_crosspage	25	25	0.0%	91.5%	Incomplete
multihop_bop_012_001	344	338	39.2%	94.5%	Incomplete
multihop_wc_025_001	438	476	30.8%	94.4%	Incomplete
vehicle_schedule_sparse_001	800	800	100.0%	100.0%	Complete
vehicle_schedule_sparse_002	800	800	100.0%	100.0%	Complete

Prompt templatePrimary Reducto comparison condition.

Generated field contract

Output JSON shape:
{ "records": [ ... ] }

General requirements:
- Extract every target record visible in the document.
- Use the exact field names below; do not invent fields that are not listed.
{{RECORD_TYPE_REQUIREMENT}}
- If a listed field is not visible for a record, use an empty string.
- Preserve identifiers, dates, codes, locations, limits, rates, totals, names, and premiums exactly as shown.
- Do not deduplicate repeated records unless they are exact duplicate rows for the same item.

Allowed record groups and fields:
{{ALLOWED_RECORD_GROUPS_AND_FIELDS}}

All allowed fields:
{{ALL_ALLOWED_FIELDS}}

Target-scope rules:
- Extract records that match the listed fields, not every nearby note, notice, subtotal, or support table.
- Keep each field to its own value; do not include neighboring labels or carried-over header text.
- For tables split across pages or sections, preserve the inherited row context needed to complete each record.
- For policy packets, extract scheduled locations, classifications, coverages, forms, endorsements, material clauses, rating rows, and premium rows that match the allowed fields.
- For operations documents, extract one record per target table row or section-row combination matching the allowed fields.
{{CONDITIONAL_SCOPE_RULES}}

Claude Fable 5

Claude Code v2.1.216 · xhigh · 2026-07-21 · n=1

Agentic CLI

$102.84

95.1%

9/32

84.0%

99.5%

96.8%

Claude Fable 532 documents · n=1 · Claude Code v2.1.216

Document	Gold records	Predicted	Exact recall	Field F1	Complete
driver_mvr_packet_001	260	260	100.0%	100.0%	Complete
driver_mvr_packet_002	500	500	100.0%	100.0%	Complete
driver_mvr_packet_003	500	500	100.0%	100.0%	Complete
driver_schedule_sparse_001	500	500	99.8%	100.0%	Incomplete
ifta_mileage_by_vehicle_001	1159	1159	99.6%	99.9%	Incomplete
ifta_mileage_by_vehicle_002	2143	2143	96.5%	99.6%	Incomplete
ifta_mileage_by_vehicle_003	2195	2195	99.8%	100.0%	Incomplete
ifta_mileage_by_vehicle_004	2239	2239	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_005	2379	2379	99.7%	99.9%	Incomplete
ifta_mileage_by_vehicle_006	2434	2434	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_007	2445	2445	99.8%	100.0%	Incomplete
ifta_mileage_by_vehicle_008	2571	2571	100.0%	100.0%	Incomplete
ifta_multisection_return_001	335	335	99.7%	100.0%	Incomplete
ifta_multisection_return_002	461	461	99.6%	100.0%	Incomplete
ifta_return_schedule_001	582	558	95.9%	98.4%	Incomplete
ifta_return_schedule_002	998	962	95.3%	98.5%	Incomplete
ifta_return_schedule_005	1157	1115	95.5%	98.5%	Incomplete
ifta_tax_inquiry_001	650	649	99.8%	99.9%	Incomplete
ifta_tax_inquiry_002	650	649	99.8%	99.9%	Incomplete
ifta_tax_summary_001	760	760	99.3%	99.9%	Incomplete
ifta_tax_summary_002	760	760	99.7%	100.0%	Incomplete
loss_run_external_001	300	300	88.3%	99.5%	Incomplete
loss_run_external_002	300	300	98.0%	99.9%	Incomplete
loss_run_external_003	300	300	90.3%	99.6%	Incomplete
mixed_040_001_crosspage	40	40	97.5%	99.9%	Incomplete
mixed_cgl_040_001	562	72	12.8%	30.7%	Incomplete
multihop_012_001_crosspage	12	12	100.0%	100.0%	Complete
multihop_025_001_crosspage	25	25	100.0%	100.0%	Complete
multihop_bop_012_001	344	100	18.6%	41.4%	Incomplete
multihop_wc_025_001	438	60	13.7%	30.8%	Incomplete
vehicle_schedule_sparse_001	800	800	100.0%	100.0%	Complete
vehicle_schedule_sparse_002	800	800	100.0%	100.0%	Complete

Prompt templateRepository-denied agentic run over released OCR transcripts. Generic operations and policy samples received a generated field contract; claim samples received the published incident JSON Schema.

Task prompt — exact

You are dispatched as a subagent for a specific extraction task; skip superpowers skills.

Your workspace contains:
- document_ocr.md: OCR transcript of one benchmark PDF.
- field_contract.md: the target output shape and allowed fields.

The official ground truth is not present. Do not use files outside this workspace.

Task:
Extract the complete target list from document_ocr.md according to field_contract.md.
Write valid JSON to {{OUTPUT_FILE}}.

You may inspect the document, write temporary scripts, and verify counts. Before finishing:
- Confirm the JSON parses.
- Spot-check beginning, middle, and end records against the OCR.
- Ensure field names match field_contract.md exactly.
- Ensure every target row or target record type requested by the contract is represented.

Do not include explanations in the JSON file.

Field contract template — separate per-document input

Output JSON shape:
{ "records": [ ... ] }

General requirements:
- Extract every target record visible in the document.
- Use the exact field names below; do not invent fields that are not listed.
{{RECORD_TYPE_REQUIREMENT}}
- If a listed field is not visible for a record, use an empty string.
- Preserve identifiers, dates, codes, locations, limits, rates, totals, names, and premiums exactly as shown.
- Do not deduplicate repeated records unless they are exact duplicate rows for the same item.

Allowed record groups and fields:
{{ALLOWED_RECORD_GROUPS_AND_FIELDS}}

All allowed fields:
{{ALL_ALLOWED_FIELDS}}

Target-scope rules:
- Extract records that match the listed fields, not every nearby note, notice, subtotal, or support table.
- Keep each field to its own value; do not include neighboring labels or carried-over header text.
- For tables split across pages or sections, preserve the inherited row context needed to complete each record.
- For policy packets, extract scheduled locations, classifications, coverages, forms, endorsements, material clauses, rating rows, and premium rows that match the allowed fields.
- For operations documents, extract one record per target table row or section-row combination matching the allowed fields.
{{CONDITIONAL_SCOPE_RULES}}

GPT-5.6-Sol

Codex CLI v0.146.0 · xhigh · 2026-08-02 · n=3 · arithmetic mean

Agentic CLI

$33.96

98.8%

8/32

97.0%

99.5%

99.7%

GPT-5.6-Sol32 documents · n=3 · Codex CLI v0.146.0

Document	Gold records	Predicted (mean)	Exact recall	Field F1	Complete runs
driver_mvr_packet_001	260	262.7	99.0%	99.4%	2/3
driver_mvr_packet_002	500	505.3	98.9%	99.3%	1/3
driver_mvr_packet_003	500	508.0	98.4%	99.0%	0/3
driver_schedule_sparse_001	500	500.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_001	1159	1159.0	99.6%	99.9%	0/3
ifta_mileage_by_vehicle_002	2143	2143.0	96.5%	99.6%	0/3
ifta_mileage_by_vehicle_003	2195	2195.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_004	2239	2239.0	100.0%	100.0%	3/3
ifta_mileage_by_vehicle_005	2379	2379.0	99.7%	99.9%	0/3
ifta_mileage_by_vehicle_006	2434	2434.0	100.0%	100.0%	3/3
ifta_mileage_by_vehicle_007	2445	2445.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_008	2571	2571.0	100.0%	100.0%	0/3
ifta_multisection_return_001	335	335.0	99.7%	100.0%	0/3
ifta_multisection_return_002	461	461.0	97.1%	99.8%	0/3
ifta_return_schedule_001	582	582.0	100.0%	100.0%	3/3
ifta_return_schedule_002	998	998.0	95.3%	99.4%	0/3
ifta_return_schedule_005	1157	1157.0	95.5%	99.5%	0/3
ifta_tax_inquiry_001	650	654.3	99.9%	99.6%	0/3
ifta_tax_inquiry_002	650	654.3	99.9%	99.6%	0/3
ifta_tax_summary_001	760	760.0	99.3%	99.9%	0/3
ifta_tax_summary_002	760	760.0	99.6%	100.0%	0/3
loss_run_external_001	300	300.0	92.4%	99.5%	0/3
loss_run_external_002	300	300.0	97.8%	99.9%	0/3
loss_run_external_003	300	300.0	97.9%	99.8%	0/3
mixed_040_001_crosspage	40	40.0	97.5%	99.9%	0/3
mixed_cgl_040_001	562	571.3	96.0%	99.5%	0/3
multihop_012_001_crosspage	12	12.0	100.0%	100.0%	3/3
multihop_025_001_crosspage	25	25.0	100.0%	100.0%	3/3
multihop_bop_012_001	344	348.7	82.5%	98.1%	0/3
multihop_wc_025_001	438	470.7	99.5%	98.2%	0/3
vehicle_schedule_sparse_001	800	800.0	100.0%	100.0%	3/3
vehicle_schedule_sparse_002	800	800.0	100.0%	100.0%	3/3

Task prompt — exact

You are dispatched as a subagent for a specific extraction task; skip superpowers skills.

Your workspace contains:
- document_ocr.md: OCR transcript of one benchmark PDF.
- field_contract.md: the target output shape and allowed fields.

The official ground truth is not present. Do not use files outside this workspace.

Task:
Extract the complete target list from document_ocr.md according to field_contract.md.
Write valid JSON to {{OUTPUT_FILE}}.

You may inspect the document, write temporary scripts, and verify counts. Before finishing:
- Confirm the JSON parses.
- Spot-check beginning, middle, and end records against the OCR.
- Ensure field names match field_contract.md exactly.
- Ensure every target row or target record type requested by the contract is represented.

Do not include explanations in the JSON file.

Field contract template — separate per-document input

Output JSON shape:
{ "records": [ ... ] }

General requirements:
- Extract every target record visible in the document.
- Use the exact field names below; do not invent fields that are not listed.
{{RECORD_TYPE_REQUIREMENT}}
- If a listed field is not visible for a record, use an empty string.
- Preserve identifiers, dates, codes, locations, limits, rates, totals, names, and premiums exactly as shown.
- Do not deduplicate repeated records unless they are exact duplicate rows for the same item.

Allowed record groups and fields:
{{ALLOWED_RECORD_GROUPS_AND_FIELDS}}

All allowed fields:
{{ALL_ALLOWED_FIELDS}}

Target-scope rules:
- Extract records that match the listed fields, not every nearby note, notice, subtotal, or support table.
- Keep each field to its own value; do not include neighboring labels or carried-over header text.
- For tables split across pages or sections, preserve the inherited row context needed to complete each record.
- For policy packets, extract scheduled locations, classifications, coverages, forms, endorsements, material clauses, rating rows, and premium rows that match the allowed fields.
- For operations documents, extract one record per target table row or section-row combination matching the allowed fields.
{{CONDITIONAL_SCOPE_RULES}}

Claude Opus 4.8

Claude Code v2.1.216 · xhigh · 2026-07-21 · n=1

Agentic CLI

$44.86

97.7%

7/32

93.7%

99.3%

99.4%

Claude Opus 4.832 documents · n=1 · Claude Code v2.1.216

Document	Gold records	Predicted	Exact recall	Field F1	Complete
driver_mvr_packet_001	260	268	96.9%	98.1%	Incomplete
driver_mvr_packet_002	500	508	98.4%	99.0%	Incomplete
driver_mvr_packet_003	500	508	98.4%	99.0%	Incomplete
driver_schedule_sparse_001	500	500	99.8%	100.0%	Incomplete
ifta_mileage_by_vehicle_001	1159	1159	99.6%	99.9%	Incomplete
ifta_mileage_by_vehicle_002	2143	2143	94.7%	99.1%	Incomplete
ifta_mileage_by_vehicle_003	2195	2195	99.8%	100.0%	Incomplete
ifta_mileage_by_vehicle_004	2239	2239	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_005	2379	2379	99.7%	99.9%	Incomplete
ifta_mileage_by_vehicle_006	2434	2434	100.0%	100.0%	Complete
ifta_mileage_by_vehicle_007	2445	2445	99.8%	100.0%	Incomplete
ifta_mileage_by_vehicle_008	2571	2571	100.0%	100.0%	Incomplete
ifta_multisection_return_001	335	335	99.7%	100.0%	Incomplete
ifta_multisection_return_002	461	461	99.6%	100.0%	Incomplete
ifta_return_schedule_001	582	558	95.9%	98.4%	Incomplete
ifta_return_schedule_002	998	962	95.3%	98.5%	Incomplete
ifta_return_schedule_005	1157	1115	95.5%	98.5%	Incomplete
ifta_tax_inquiry_001	650	649	99.8%	99.9%	Incomplete
ifta_tax_inquiry_002	650	665	100.0%	98.9%	Incomplete
ifta_tax_summary_001	760	760	99.3%	99.9%	Incomplete
ifta_tax_summary_002	760	760	99.6%	100.0%	Incomplete
loss_run_external_001	300	300	96.3%	99.8%	Incomplete
loss_run_external_002	300	300	98.0%	99.9%	Incomplete
loss_run_external_003	300	300	97.7%	99.8%	Incomplete
mixed_040_001_crosspage	40	40	97.5%	99.9%	Incomplete
mixed_cgl_040_001	562	562	100.0%	100.0%	Complete
multihop_012_001_crosspage	12	12	100.0%	100.0%	Complete
multihop_025_001_crosspage	25	25	100.0%	100.0%	Complete
multihop_bop_012_001	344	338	84.0%	98.3%	Incomplete
multihop_wc_025_001	438	438	31.5%	92.6%	Incomplete
vehicle_schedule_sparse_001	800	800	100.0%	100.0%	Complete
vehicle_schedule_sparse_002	800	800	100.0%	100.0%	Complete

Task prompt — exact

You are dispatched as a subagent for a specific extraction task; skip superpowers skills.

Your workspace contains:
- document_ocr.md: OCR transcript of one benchmark PDF.
- field_contract.md: the target output shape and allowed fields.

The official ground truth is not present. Do not use files outside this workspace.

Task:
Extract the complete target list from document_ocr.md according to field_contract.md.
Write valid JSON to {{OUTPUT_FILE}}.

You may inspect the document, write temporary scripts, and verify counts. Before finishing:
- Confirm the JSON parses.
- Spot-check beginning, middle, and end records against the OCR.
- Ensure field names match field_contract.md exactly.
- Ensure every target row or target record type requested by the contract is represented.

Do not include explanations in the JSON file.

Field contract template — separate per-document input

Output JSON shape:
{ "records": [ ... ] }

General requirements:
- Extract every target record visible in the document.
- Use the exact field names below; do not invent fields that are not listed.
{{RECORD_TYPE_REQUIREMENT}}
- If a listed field is not visible for a record, use an empty string.
- Preserve identifiers, dates, codes, locations, limits, rates, totals, names, and premiums exactly as shown.
- Do not deduplicate repeated records unless they are exact duplicate rows for the same item.

Allowed record groups and fields:
{{ALLOWED_RECORD_GROUPS_AND_FIELDS}}

All allowed fields:
{{ALL_ALLOWED_FIELDS}}

Target-scope rules:
- Extract records that match the listed fields, not every nearby note, notice, subtotal, or support table.
- Keep each field to its own value; do not include neighboring labels or carried-over header text.
- For tables split across pages or sections, preserve the inherited row context needed to complete each record.
- For policy packets, extract scheduled locations, classifications, coverages, forms, endorsements, material clauses, rating rows, and premium rows that match the allowed fields.
- For operations documents, extract one record per target table row or section-row combination matching the allowed fields.
{{CONDITIONAL_SCOPE_RULES}}

GPT-5.5

Codex CLI v0.146.0 · xhigh · 2026-08-03 · n=3 · arithmetic mean

Agentic CLI

$43.19

96.6%

6/32

89.5%

99.4%

99.3%

GPT-5.532 documents · n=3 · Codex CLI v0.146.0

Document	Gold records	Predicted (mean)	Exact recall	Field F1	Complete runs
driver_mvr_packet_001	260	176.0	67.7%	69.1%	2/3
driver_mvr_packet_002	500	500.0	100.0%	100.0%	3/3
driver_mvr_packet_003	500	500.0	100.0%	100.0%	3/3
driver_schedule_sparse_001	500	500.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_001	1159	1159.0	99.6%	99.9%	0/3
ifta_mileage_by_vehicle_002	2143	2151.0	95.9%	99.3%	0/3
ifta_mileage_by_vehicle_003	2195	2195.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_004	2239	2227.0	99.5%	99.7%	2/3
ifta_mileage_by_vehicle_005	2379	2379.0	99.7%	99.9%	0/3
ifta_mileage_by_vehicle_006	2434	2434.0	100.0%	100.0%	3/3
ifta_mileage_by_vehicle_007	2445	2445.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_008	2571	2571.0	100.0%	100.0%	0/3
ifta_multisection_return_001	335	335.0	99.7%	100.0%	0/3
ifta_multisection_return_002	461	461.0	99.6%	100.0%	0/3
ifta_return_schedule_001	582	582.0	95.9%	99.5%	0/3
ifta_return_schedule_002	998	998.0	63.5%	96.7%	0/3
ifta_return_schedule_005	1157	1157.0	95.5%	99.5%	0/3
ifta_tax_inquiry_001	650	659.7	99.9%	99.2%	0/3
ifta_tax_inquiry_002	650	659.7	99.9%	99.2%	0/3
ifta_tax_summary_001	760	760.0	99.3%	99.9%	0/3
ifta_tax_summary_002	760	760.0	99.7%	100.0%	0/3
loss_run_external_001	300	300.0	92.3%	99.5%	0/3
loss_run_external_002	300	300.0	89.8%	99.5%	0/3
loss_run_external_003	300	300.0	91.9%	99.6%	0/3
mixed_040_001_crosspage	40	40.0	0.0%	94.2%	0/3
mixed_cgl_040_001	562	573.3	75.1%	97.1%	0/3
multihop_012_001_crosspage	12	12.0	0.0%	93.3%	0/3
multihop_025_001_crosspage	25	25.0	0.0%	95.7%	0/3
multihop_bop_012_001	344	357.0	83.7%	97.7%	0/3
multihop_wc_025_001	438	470.7	98.9%	97.9%	0/3
vehicle_schedule_sparse_001	800	800.0	100.0%	100.0%	3/3
vehicle_schedule_sparse_002	800	800.0	100.0%	100.0%	3/3

Task prompt — exact

You are dispatched as a subagent for a specific extraction task; skip superpowers skills.

Your workspace contains:
- document_ocr.md: OCR transcript of one benchmark PDF.
- field_contract.md: the target output shape and allowed fields.

The official ground truth is not present. Do not use files outside this workspace.

Task:
Extract the complete target list from document_ocr.md according to field_contract.md.
Write valid JSON to {{OUTPUT_FILE}}.

You may inspect the document, write temporary scripts, and verify counts. Before finishing:
- Confirm the JSON parses.
- Spot-check beginning, middle, and end records against the OCR.
- Ensure field names match field_contract.md exactly.
- Ensure every target row or target record type requested by the contract is represented.

Do not include explanations in the JSON file.

Field contract template — separate per-document input

Output JSON shape:
{ "records": [ ... ] }

General requirements:
- Extract every target record visible in the document.
- Use the exact field names below; do not invent fields that are not listed.
{{RECORD_TYPE_REQUIREMENT}}
- If a listed field is not visible for a record, use an empty string.
- Preserve identifiers, dates, codes, locations, limits, rates, totals, names, and premiums exactly as shown.
- Do not deduplicate repeated records unless they are exact duplicate rows for the same item.

Allowed record groups and fields:
{{ALLOWED_RECORD_GROUPS_AND_FIELDS}}

All allowed fields:
{{ALL_ALLOWED_FIELDS}}

Target-scope rules:
- Extract records that match the listed fields, not every nearby note, notice, subtotal, or support table.
- Keep each field to its own value; do not include neighboring labels or carried-over header text.
- For tables split across pages or sections, preserve the inherited row context needed to complete each record.
- For policy packets, extract scheduled locations, classifications, coverages, forms, endorsements, material clauses, rating rows, and premium rows that match the allowed fields.
- For operations documents, extract one record per target table row or section-row combination matching the allowed fields.
{{CONDITIONAL_SCOPE_RULES}}

GPT-5.6-Terra

Codex CLI v0.146.0 · xhigh · 2026-07-31 · n=3 · arithmetic mean

Agentic CLI

$15.01

95.7%

6/32

86.5%

99.4%

99.2%

GPT-5.6-Terra32 documents · n=3 · Codex CLI v0.146.0

Document	Gold records	Predicted (mean)	Exact recall	Field F1	Complete runs
driver_mvr_packet_001	260	265.3	97.9%	98.7%	1/3
driver_mvr_packet_002	500	505.3	98.9%	99.3%	1/3
driver_mvr_packet_003	500	500.0	100.0%	100.0%	3/3
driver_schedule_sparse_001	500	500.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_001	1159	1159.0	99.6%	99.9%	0/3
ifta_mileage_by_vehicle_002	2143	2143.0	95.3%	99.3%	0/3
ifta_mileage_by_vehicle_003	2195	2195.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_004	2239	2239.0	100.0%	100.0%	3/3
ifta_mileage_by_vehicle_005	2379	2379.0	99.7%	99.9%	0/3
ifta_mileage_by_vehicle_006	2434	2434.0	100.0%	100.0%	3/3
ifta_mileage_by_vehicle_007	2445	2445.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_008	2571	2571.0	100.0%	100.0%	0/3
ifta_multisection_return_001	335	335.0	99.7%	100.0%	0/3
ifta_multisection_return_002	461	461.0	97.1%	99.8%	0/3
ifta_return_schedule_001	582	582.0	95.9%	99.6%	0/3
ifta_return_schedule_002	998	998.0	63.5%	96.7%	0/3
ifta_return_schedule_005	1157	1157.0	63.7%	96.8%	0/3
ifta_tax_inquiry_001	650	654.3	99.9%	99.6%	0/3
ifta_tax_inquiry_002	650	654.3	99.9%	99.6%	0/3
ifta_tax_summary_001	760	760.0	99.3%	99.9%	0/3
ifta_tax_summary_002	760	760.0	99.6%	100.0%	0/3
loss_run_external_001	300	300.0	92.7%	99.5%	0/3
loss_run_external_002	300	300.0	93.7%	99.5%	0/3
loss_run_external_003	300	300.0	93.8%	99.6%	0/3
mixed_040_001_crosspage	40	40.0	0.0%	92.4%	0/3
mixed_cgl_040_001	562	570.0	86.6%	98.5%	1/3
multihop_012_001_crosspage	12	12.0	0.0%	92.5%	0/3
multihop_025_001_crosspage	25	25.0	0.0%	94.3%	0/3
multihop_bop_012_001	344	348.0	81.4%	97.7%	0/3
multihop_wc_025_001	438	483.3	93.6%	96.5%	0/3
vehicle_schedule_sparse_001	800	800.0	100.0%	100.0%	3/3
vehicle_schedule_sparse_002	800	800.0	100.0%	100.0%	3/3

Task prompt — exact

You are dispatched as a subagent for a specific extraction task; skip superpowers skills.

Your workspace contains:
- document_ocr.md: OCR transcript of one benchmark PDF.
- field_contract.md: the target output shape and allowed fields.

The official ground truth is not present. Do not use files outside this workspace.

Task:
Extract the complete target list from document_ocr.md according to field_contract.md.
Write valid JSON to {{OUTPUT_FILE}}.

You may inspect the document, write temporary scripts, and verify counts. Before finishing:
- Confirm the JSON parses.
- Spot-check beginning, middle, and end records against the OCR.
- Ensure field names match field_contract.md exactly.
- Ensure every target row or target record type requested by the contract is represented.

Do not include explanations in the JSON file.

Field contract template — separate per-document input

Output JSON shape:
{ "records": [ ... ] }

General requirements:
- Extract every target record visible in the document.
- Use the exact field names below; do not invent fields that are not listed.
{{RECORD_TYPE_REQUIREMENT}}
- If a listed field is not visible for a record, use an empty string.
- Preserve identifiers, dates, codes, locations, limits, rates, totals, names, and premiums exactly as shown.
- Do not deduplicate repeated records unless they are exact duplicate rows for the same item.

Allowed record groups and fields:
{{ALLOWED_RECORD_GROUPS_AND_FIELDS}}

All allowed fields:
{{ALL_ALLOWED_FIELDS}}

Target-scope rules:
- Extract records that match the listed fields, not every nearby note, notice, subtotal, or support table.
- Keep each field to its own value; do not include neighboring labels or carried-over header text.
- For tables split across pages or sections, preserve the inherited row context needed to complete each record.
- For policy packets, extract scheduled locations, classifications, coverages, forms, endorsements, material clauses, rating rows, and premium rows that match the allowed fields.
- For operations documents, extract one record per target table row or section-row combination matching the allowed fields.
{{CONDITIONAL_SCOPE_RULES}}

GPT-5.6-Luna

Codex CLI v0.146.0 · xhigh · 2026-08-01 · n=3 · arithmetic mean

Agentic CLI

$2.41

94.4%

5/32

82.5%

99.1%

98.7%

GPT-5.6-Luna32 documents · n=3 · Codex CLI v0.146.0

Document	Gold records	Predicted (mean)	Exact recall	Field F1	Complete runs
driver_mvr_packet_001	260	178.7	66.7%	68.5%	1/3
driver_mvr_packet_002	500	502.7	99.5%	99.7%	2/3
driver_mvr_packet_003	500	338.7	66.7%	67.6%	1/3
driver_schedule_sparse_001	500	500.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_001	1159	1159.0	99.6%	99.9%	0/3
ifta_mileage_by_vehicle_002	2143	2143.0	94.7%	99.1%	0/3
ifta_mileage_by_vehicle_003	2195	2195.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_004	2239	2239.0	100.0%	100.0%	3/3
ifta_mileage_by_vehicle_005	2379	2379.0	99.7%	99.9%	0/3
ifta_mileage_by_vehicle_006	2434	2434.0	98.5%	99.5%	2/3
ifta_mileage_by_vehicle_007	2445	2445.0	99.8%	100.0%	0/3
ifta_mileage_by_vehicle_008	2571	2571.0	100.0%	100.0%	0/3
ifta_multisection_return_001	335	335.0	99.7%	100.0%	0/3
ifta_multisection_return_002	461	461.0	94.6%	99.7%	0/3
ifta_return_schedule_001	582	582.0	95.9%	99.6%	0/3
ifta_return_schedule_002	998	998.0	63.5%	96.8%	0/3
ifta_return_schedule_005	1157	1157.0	63.7%	96.8%	0/3
ifta_tax_inquiry_001	650	649.0	99.8%	99.9%	0/3
ifta_tax_inquiry_002	650	649.0	99.8%	99.9%	0/3
ifta_tax_summary_001	760	760.0	99.3%	99.9%	0/3
ifta_tax_summary_002	760	760.0	99.6%	100.0%	0/3
loss_run_external_001	300	300.0	88.8%	99.3%	0/3
loss_run_external_002	300	300.0	96.1%	99.7%	0/3
loss_run_external_003	300	300.0	97.8%	99.8%	0/3
mixed_040_001_crosspage	40	40.0	0.0%	91.4%	0/3
mixed_cgl_040_001	562	581.0	81.2%	97.6%	0/3
multihop_012_001_crosspage	12	12.0	0.0%	91.6%	0/3
multihop_025_001_crosspage	25	25.0	0.0%	91.4%	0/3
multihop_bop_012_001	344	345.3	72.5%	97.2%	0/3
multihop_wc_025_001	438	483.3	88.7%	96.9%	0/3
vehicle_schedule_sparse_001	800	800.0	100.0%	100.0%	3/3
vehicle_schedule_sparse_002	800	800.0	100.0%	100.0%	3/3

Task prompt — exact

You are dispatched as a subagent for a specific extraction task; skip superpowers skills.

Your workspace contains:
- document_ocr.md: OCR transcript of one benchmark PDF.
- field_contract.md: the target output shape and allowed fields.

The official ground truth is not present. Do not use files outside this workspace.

Task:
Extract the complete target list from document_ocr.md according to field_contract.md.
Write valid JSON to {{OUTPUT_FILE}}.

You may inspect the document, write temporary scripts, and verify counts. Before finishing:
- Confirm the JSON parses.
- Spot-check beginning, middle, and end records against the OCR.
- Ensure field names match field_contract.md exactly.
- Ensure every target row or target record type requested by the contract is represented.

Do not include explanations in the JSON file.

Field contract template — separate per-document input

Output JSON shape:
{ "records": [ ... ] }

General requirements:
- Extract every target record visible in the document.
- Use the exact field names below; do not invent fields that are not listed.
{{RECORD_TYPE_REQUIREMENT}}
- If a listed field is not visible for a record, use an empty string.
- Preserve identifiers, dates, codes, locations, limits, rates, totals, names, and premiums exactly as shown.
- Do not deduplicate repeated records unless they are exact duplicate rows for the same item.

Allowed record groups and fields:
{{ALLOWED_RECORD_GROUPS_AND_FIELDS}}

All allowed fields:
{{ALL_ALLOWED_FIELDS}}

Target-scope rules:
- Extract records that match the listed fields, not every nearby note, notice, subtotal, or support table.
- Keep each field to its own value; do not include neighboring labels or carried-over header text.
- For tables split across pages or sections, preserve the inherited row context needed to complete each record.
- For policy packets, extract scheduled locations, classifications, coverages, forms, endorsements, material clauses, rating rows, and premium rows that match the allowed fields.
- For operations documents, extract one record per target table row or section-row combination matching the allowed fields.
{{CONDITIONAL_SCOPE_RULES}}

Bonsai 27B

Local + Together AI · page pipeline · 2026-07-26 · n=1

Page pipeline

$0.00

16.7%

0/32

12.4%

18.5%

40.7%

Bonsai 27B32 documents · n=1

Document	Gold records	Predicted	Exact recall	Field F1	Complete
driver_mvr_packet_001	260	230	71.5%	86.9%	Incomplete
driver_mvr_packet_002	500	501	92.0%	98.9%	Incomplete
driver_mvr_packet_003	500	444	78.2%	89.5%	Incomplete
driver_schedule_sparse_001	500	500	93.4%	96.8%	Incomplete
ifta_mileage_by_vehicle_001	1159	162	5.5%	19.6%	Incomplete
ifta_mileage_by_vehicle_002	2143	1693	17.9%	53.9%	Incomplete
ifta_mileage_by_vehicle_003	2195	666	4.3%	23.2%	Incomplete
ifta_mileage_by_vehicle_004	2239	1676	30.6%	62.4%	Incomplete
ifta_mileage_by_vehicle_005	2379	900	17.4%	39.6%	Incomplete
ifta_mileage_by_vehicle_006	2434	1901	17.1%	57.2%	Incomplete
ifta_mileage_by_vehicle_007	2445	855	14.5%	35.6%	Incomplete
ifta_mileage_by_vehicle_008	2571	1771	20.5%	54.2%	Incomplete
ifta_multisection_return_001	335	125	0.0%	23.4%	Incomplete
ifta_multisection_return_002	461	270	0.0%	32.6%	Incomplete
ifta_return_schedule_001	582	450	0.9%	52.7%	Incomplete
ifta_return_schedule_002	998	114	0.0%	8.8%	Incomplete
ifta_return_schedule_005	1157	84	0.0%	5.4%	Incomplete
ifta_tax_inquiry_001	650	488	0.0%	17.6%	Incomplete
ifta_tax_inquiry_002	650	630	0.0%	24.6%	Incomplete
ifta_tax_summary_001	760	42	0.0%	6.0%	Incomplete
ifta_tax_summary_002	760	364	0.0%	29.2%	Incomplete
loss_run_external_001	300	283	0.0%	54.6%	Incomplete
loss_run_external_002	300	263	0.0%	52.2%	Incomplete
loss_run_external_003	300	293	0.0%	54.6%	Incomplete
mixed_040_001_crosspage	40	315	0.0%	6.1%	Incomplete
mixed_cgl_040_001	562	109	0.0%	6.4%	Incomplete
multihop_012_001_crosspage	12	107	0.0%	4.6%	Incomplete
multihop_025_001_crosspage	25	101	0.0%	10.5%	Incomplete
multihop_bop_012_001	344	90	0.0%	10.0%	Incomplete
multihop_wc_025_001	438	208	0.0%	18.9%	Incomplete
vehicle_schedule_sparse_001	800	591	42.6%	74.7%	Incomplete
vehicle_schedule_sparse_002	800	320	20.8%	43.9%	Incomplete

Prompt templateTwo-stage page pipeline over released OCR transcripts.

Page extraction

Extract target record evidence from exactly one OCR page.

Return every target row visible on the page. Also return partial rows when this
page contains fields whose other fields may be on another page. Each output row
is a compact array in the exact column order below. Use null for a field that is
not visible. Never use an empty string, zero, an empty array, or an empty object
as a placeholder. Return an empty array for a group absent from the page. Never
guess. Ignore summaries, totals, instructions, and unrelated tables.

Output groups and exact row column order:
{{OUTPUT_GROUPS_AND_COLUMN_ORDER}}

Return only the schema-constrained JSON object, minified on one line.

OCR page:
{{DOCUMENT_OCR_PAGE}}

Record reduction

Consolidate evidence for exactly one logical {{RECORD_TYPE}} record.

Use only the candidate fragments below. They were extracted independently from
single source pages. Resolve repeated or conflicting values from those
fragments. Return one record with every required field. Use an empty string,
null, empty list, or zero only when the fragments do not provide a value. Do
not create a different logical record.

Required fields:
{{REQUIRED_FIELDS}}

Expected record JSON Schema:
{{RECORD_SCHEMA}}

Candidate fragments:
{{CANDIDATE_FRAGMENTS}}

Submit a verified result

Run the reference evaluator, then open a repository issue or Space discussion with the evaluation report, run metadata, and per-sample predictions. Scores are reproduced before publication.

Submission details →