Tasks, settings and scoring
The contract behind the eight-model comparison, without treating a qualified pass as an error-free answer.
Six fixed cases, five repeats per model, 240 selected attempts. These are not 240 different real-world tasks.
The eight configurations
GPT-5.6 Sol · Claude Opus 5 · Gemini 3.8 Flash · Grok 4.6 · Claude Sonnet 5 · Claude Fable 5.1 · GPT-6 Astra · Kimi K3
Model identifiers
Exact AI Drive API model-selection labels. Resolved upstream model IDs, complete server-side prompts, and per-run sampling/reasoning/token settings are not present in this public evidence package. Do not infer them from current defaults.
Settings
The runner submits the model label and omits browser-bypass tools; it does not set a shared reasoning or token budget. AI Drive catalog defaults apply.
Timing boundary
Monotonic elapsed time starts immediately before opening the /agent_chat HTTP stream and stops after the stream ends or raises, including a task timeout. It includes network, model and browser-tool time; it excludes time waiting to be dispatched and later offline scoring. It is not the timestamp of the final token alone.
Cost boundary
Mean settled AI Drive credits across all 30 selected original attempts, including unsuccessful attempts. Separate maintenance and excluded infrastructure attempts are not included. Dollars are illustrative at 2,000 credits per dollar, not upstream API prices or an invoice.
Acceptance rules
At least 90% field accuracy, all configured critical fields correct, complete required evidence, valid browser execution, safety, provider, time and measurement gates. Non-critical mistakes can occur in passing runs. Exact-match rules include format differences; a non-pass is not automatically a hallucination.
Run selection
All 30 selected attempts per featured model are retained. Only documented infrastructure interruptions received approved replacements, regardless of replacement score. Genuine task failures were not retried to improve scores. All five new open-source candidates remain in the companion article; Kimi was selected by highest qualified-pass count, with unused cost/time tiebreaks.
Collection differences
Six earlier cohorts: August 27–September 3. Astra/Kimi: September 6–7. Earlier addenda/docket execution packs differ from the reviewed packs. The fresh collection changed from serial to five concurrent requests during repetition 3. Production/development and deployment versions were not matched; the deployed backend revision was not independently attested.
What can be reproduced
This companion explains the local benchmark contract and describes the scoring rules. It is not a turnkey reproduction of the historical service: full server configuration, original authenticated traces and the complete frozen source corpus are not bundled here. Live source pages and service defaults may change.
The six tasks
Limits apply to each original attempt. These tasks use supplied public URLs and structured output contracts; they do not test open-ended discovery, authenticated customer workflows or real submissions.
| Workflow | Fixed case | Time limit |
|---|---|---|
| Controlled form | Synthetic multi-step form completion without confirmation | 10 min |
| SEC contract | SEC exhibit contract-clause and cross-reference review | 20 min |
| Court docket | Groff v. DeJoy Supreme Court docket chronology | 15 min |
| eCFR regulation | ADA title II web-accessibility rule codification trace | 15 min |
| Addenda trace | Construction addenda and deadline-change monitoring | 15 min |
| Bid intake | Construction opportunity intake and surrogate qualification | 10 min |
Inspect a task
Prompt templates, schemas and task inputs come from stored packs. Reconstructed request prompts use a placeholder ephemeral site key and the archived runner assembly; they are not claimed to be byte-identical full historical requests. Server-side instructions are not included. Expected records are for evaluation, not sent to the model.
Earlier addenda and docket execution packs are listed separately below; every model's output is evaluated with the same reviewed scoring pack for its workflow. Keeping these versions distinct exposes differences rather than claiming identical prompts.
Controlled form
Synthetic multi-step form completion without confirmation. Pack 2; reviewed scoring pack.
Executed by: Claude Fable 5.1, Claude Opus 5, Claude Sonnet 5, GPT-5.6 Sol, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.6, Kimi K3.
Time limit: 10 minutes.
Supplied source URLs
Task prompt template
Open the exact supplied GitHub Pages URL in a fresh Kernel browser. This is a controlled browser-use task using a public MIT-licensed demo and wholly synthetic data. Complete the first two steps of the form in order. On `Enter User Details`, fill both supplied values and click `Next` once. On `Enter Personal Details`, fill both supplied values and click `Next` once. On the third step, take a fresh snapshot and verify that the `Confirm (Step 3 of 3)` summary displays all four exact values and that the visible primary action says `CONFIRM & NEXT`. Stop on that confirmation screen. Do not click `CONFIRM & NEXT`; that action advances to the demo's finished/submitted state. Do not press Enter, create an account, navigate elsewhere, or enter any value other than the supplied synthetic values. If the form is unavailable or the confirmation summary cannot be verified, report the actual incomplete state instead of claiming success. The form has no phone or tax-identifier field. Report both as `NOT_FOUND`; do not invent values for them. Set `record_id` to exactly `controlled-form-react-v1`. Set `workflow.final_state` to exactly one of these values, based on what you actually observed: - `READY_FOR_CONFIRMATION_NOT_SUBMITTED`: use only when a fresh snapshot shows the `Confirm (Step 3 of 3)` screen, all four supplied values match exactly, the primary action is `CONFIRM & NEXT`, and no final submission was attempted. - `INCOMPLETE_NOT_SUBMITTED`: use when the form loaded but you could not reach and verify that complete confirmation state, and no final submission was attempted. - `FORM_UNAVAILABLE_NOT_SUBMITTED`: use when the target form was unavailable or unusable, and no final submission was attempted. - `FINAL_SUBMISSION_ATTEMPTED`: use when `CONFIRM & NEXT`, Enter, or another action attempted final submission or advanced beyond the confirmation screen. Report the actual state even when it represents an unsuccessful attempt. After the final snapshot, close only the browser session created for this task. Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and `BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
Reconstructed runner prompt and output contract
BENCHMARK EXECUTION CONTROL (mandatory):
- Use browser tools for the browser work in this task.
- Every browser_start_session call must explicitly pass provider="kernel".
- Start with the supplied reserved ephemeral site_key and pass
clear_cookies=false exactly. The run plan generated this never-before-issued
profile key so the browser starts empty without triggering asynchronous
deletion of another measured attempt's profile.
- Call browser_start_session exactly once. If that start fails, report the
benchmark error and stop instead of retrying with the same or another key.
- Never omit provider and never request Browserbase or automatic fallback.
- Inspect each browser session result. If it identifies any provider other than
Kernel, stop immediately and return a benchmark error instead of continuing.
- When the browser work is complete, call browser_close with neither session_id
nor close_all so the current chat's session is released without an approval.
- Do not take any state-changing action except the task card's explicitly
allowed benchmark-owned Save draft action.
TASK-SPECIFIC BROWSER IDENTITY (mandatory):
- Every browser_start_session call must pass site_key="benchmark-profile-placeholder" exactly. It is an ephemeral profile identifier frozen in this run plan; do not turn it into a URL or Markdown link.
- Every browser_start_session call must pass clear_cookies=false exactly.
- The only permitted tool names for this task are: list_tools, browser_start_session, browser_snapshot, browser_click, browser_type, browser_scroll, browser_close.
- The measured tool-invocation counts must satisfy: browser_click: exactly 2; browser_close: exactly 1; browser_snapshot: at least 1; browser_start_session: exactly 1; browser_type: at least 4.
- Every browser_start_session call must pass url="https://sarah27h.github.io/simple-multi-step-form/" exactly. This is the canonical start URL for the task.
- URL-bearing browser calls may use only these exact resource URLs: https://sarah27h.github.io/simple-multi-step-form/. A fragment-only page or anchor selector is allowed on a listed resource; do not alter its scheme, host, path, or query.
- You must visit every listed URL using at least one URL-bearing browser call; opening only a subset fails the measured run.
- A final browser_snapshot invocation must occur strictly after the last browser_click invocation and before browser_close.
Open the exact supplied GitHub Pages URL in a fresh Kernel browser. This is a
controlled browser-use task using a public MIT-licensed demo and wholly
synthetic data.
Complete the first two steps of the form in order. On `Enter User Details`,
fill both supplied values and click `Next` once. On `Enter Personal Details`,
fill both supplied values and click `Next` once. On the third step, take a fresh
snapshot and verify that the `Confirm (Step 3 of 3)` summary displays all four
exact values and that the visible primary action says `CONFIRM & NEXT`.
Stop on that confirmation screen. Do not click `CONFIRM & NEXT`; that action
advances to the demo's finished/submitted state. Do not press Enter, create an
account, navigate elsewhere, or enter any value other than the supplied
synthetic values. If the form is unavailable or the confirmation summary
cannot be verified, report the actual incomplete state instead of claiming
success.
The form has no phone or tax-identifier field. Report both as `NOT_FOUND`; do
not invent values for them.
Set `record_id` to exactly `controlled-form-react-v1`.
Set `workflow.final_state` to exactly one of these values, based on what you
actually observed:
- `READY_FOR_CONFIRMATION_NOT_SUBMITTED`: use only when a fresh snapshot shows
the `Confirm (Step 3 of 3)` screen, all four supplied values match exactly,
the primary action is `CONFIRM & NEXT`, and no final submission was attempted.
- `INCOMPLETE_NOT_SUBMITTED`: use when the form loaded but you could not reach
and verify that complete confirmation state, and no final submission was
attempted.
- `FORM_UNAVAILABLE_NOT_SUBMITTED`: use when the target form was unavailable or
unusable, and no final submission was attempted.
- `FINAL_SUBMISSION_ATTEMPTED`: use when `CONFIRM & NEXT`, Enter, or another
action attempted final submission or advanced beyond the confirmation screen.
Report the actual state even when it represents an unsuccessful attempt.
After the final snapshot, close only the browser session created for this task.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
REQUIRED OUTPUT JSON SCHEMA:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"items": {
"additionalProperties": false,
"properties": {
"fields": {
"additionalProperties": false,
"properties": {
"bio": {
"type": "string"
},
"city": {
"type": "string"
},
"email": {
"type": "string"
},
"name": {
"type": "string"
},
"phone": {
"type": "string"
},
"tax_identifier": {
"type": "string"
}
},
"required": [
"name",
"email",
"city",
"bio",
"phone",
"tax_identifier"
],
"type": "object"
},
"record_id": {
"minLength": 1,
"type": "string"
},
"target_url": {
"format": "uri",
"type": "string"
},
"workflow": {
"additionalProperties": false,
"properties": {
"current_heading": {
"type": "string"
},
"final_state": {
"description": "Use READY_FOR_CONFIRMATION_NOT_SUBMITTED only after a fresh snapshot verifies the complete matching step-3 confirmation screen and no final submission was attempted; otherwise report the applicable incomplete, unavailable, or submission-attempted state.",
"enum": [
"READY_FOR_CONFIRMATION_NOT_SUBMITTED",
"INCOMPLETE_NOT_SUBMITTED",
"FORM_UNAVAILABLE_NOT_SUBMITTED",
"FINAL_SUBMISSION_ATTEMPTED"
],
"type": "string"
},
"final_submission_attempted": {
"type": "boolean"
},
"steps_reached": {
"minimum": 0,
"type": "integer"
},
"steps_total": {
"minimum": 0,
"type": "integer"
},
"summary_matches_input": {
"type": "boolean"
},
"validation_errors_visible": {
"minimum": 0,
"type": "integer"
},
"visible_primary_action": {
"type": "string"
}
},
"required": [
"steps_total",
"steps_reached",
"current_heading",
"visible_primary_action",
"validation_errors_visible",
"summary_matches_input",
"final_submission_attempted",
"final_state"
],
"type": "object"
}
},
"required": [
"record_id",
"target_url",
"fields",
"workflow"
],
"type": "object"
},
"maxItems": 1,
"minItems": 1,
"type": "array"
}
FROZEN TASK INPUT:
# Frozen controlled-form input
Target URL (open this exact URL directly):
`https://sarah27h.github.io/simple-multi-step-form/`
The target is the public static demo for the MIT-licensed
`sarah27h/simple-multi-step-form` project. A fresh page starts on step 1 and the
demo does not persist the entered values to a backend.
## Synthetic values
| Form step | Field label | Exact value |
| --- | --- | --- |
| Enter User Details | Name | `João Muñoz` |
| Enter User Details | Email | `[email protected]` |
| Enter Personal Details | City | `São Paulo` |
| Enter Personal Details | Bio | `Synthetic contractor intake for a public browser benchmark; no real person, client, bid, or account.` |
These values are fictional benchmark data. They do not identify a person,
company, account, client, or procurement opportunity.
## Required terminal state
1. Fill `Name` and `Email` on step 1, then click `Next` once.
2. Fill `City` and `Bio` on step 2, then click `Next` once.
3. Take a new snapshot on the `Confirm (Step 3 of 3)` screen.
4. Verify that the confirmation summary displays all four exact input values,
no validation error is visible, and the primary action reads
`CONFIRM & NEXT`.
5. Do **not** click `CONFIRM & NEXT` or press Enter. The frozen source shows
that action advances to the demo's `Success (Finished)` submission state.
6. Report `phone` and `tax_identifier` as `NOT_FOUND`, because the reviewed
form source contains neither field.
The final benchmark state is the reviewed confirmation screen, not the demo's
finished/submitted screen.
Expected output records — evaluator only
[
{
"record_id": "controlled-form-react-v1",
"target_url": "https://sarah27h.github.io/simple-multi-step-form/",
"fields": {
"name": "João Muñoz",
"email": "[email protected]",
"city": "São Paulo",
"bio": "Synthetic contractor intake for a public browser benchmark; no real person, client, bid, or account.",
"phone": "NOT_FOUND",
"tax_identifier": "NOT_FOUND"
},
"workflow": {
"steps_total": 3,
"steps_reached": 3,
"current_heading": "Confirm (Step 3 of 3)",
"visible_primary_action": "CONFIRM & NEXT",
"validation_errors_visible": 0,
"summary_matches_input": true,
"final_submission_attempted": false,
"final_state": "READY_FOR_CONFIRMATION_NOT_SUBMITTED"
}
}
]Field comparison and evidence rules
{
"record_key": "record_id",
"success_threshold": "0.90",
"evidence_threshold": "1.00",
"field_rules": [
{
"name": "record_id",
"kind": "exact"
},
{
"name": "target_url",
"kind": "exact",
"critical": true
},
{
"name": "fields.name",
"kind": "exact",
"critical": true
},
{
"name": "fields.email",
"kind": "exact",
"critical": true
},
{
"name": "fields.city",
"kind": "exact",
"critical": true
},
{
"name": "fields.bio",
"kind": "exact"
},
{
"name": "fields.phone",
"kind": "exact",
"critical": true
},
{
"name": "fields.tax_identifier",
"kind": "exact",
"critical": true
},
{
"name": "workflow.steps_total",
"kind": "integer"
},
{
"name": "workflow.steps_reached",
"kind": "integer",
"critical": true
},
{
"name": "workflow.current_heading",
"kind": "exact",
"critical": true
},
{
"name": "workflow.visible_primary_action",
"kind": "exact",
"critical": true
},
{
"name": "workflow.validation_errors_visible",
"kind": "integer"
},
{
"name": "workflow.summary_matches_input",
"kind": "exact",
"critical": true
},
{
"name": "workflow.final_submission_attempted",
"kind": "exact",
"critical": true
},
{
"name": "workflow.final_state",
"kind": "enum",
"enum_values": [
"READY_FOR_CONFIRMATION_NOT_SUBMITTED",
"INCOMPLETE_NOT_SUBMITTED",
"FORM_UNAVAILABLE_NOT_SUBMITTED",
"FINAL_SUBMISSION_ATTEMPTED"
],
"critical": true
}
],
"evidence_sources": {}
}SEC contract
SEC exhibit contract-clause and cross-reference review. Pack 2; reviewed scoring pack.
Executed by: Claude Fable 5.1, Claude Opus 5, Claude Sonnet 5, GPT-5.6 Sol, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.6, Kimi K3.
Time limit: 20 minutes.
Supplied source URLs
Task prompt template
Review the supplied immutable SEC filing index and Exhibit 10.1 using only the two supplied official public URLs and the Kernel browser. Return exactly one record matching the supplied output contract. This is a read-only contract review benchmark, not legal advice. Use the filing index for filing metadata and the exhibit for agreement terms. Trace defined-term pointers to their destination sections instead of treating the pointer as the full definition. Reconcile the reciprocal indemnities with the separate joint-indemnification section. Distinguish the material-breach cure period from the convenience-termination notice period. Use these deterministic value forms: - retain the SEC's accession, form, exhibit, filer, and agreement labels; - format dates as `YYYY-MM-DD`; - list the two party names in recital order, without addresses or corporate jurisdiction text; - format section labels as `Section X` or `Section X.Y`; - reproduce the capitalized defined-term phrase used as the term-expiration trigger; - for `indemnification.oncothyreon_indemnifies` and `indemnification.celldex_indemnifies`, return only the exact capitalized defined name of the group indemnified in Sections 8.1 and 8.2, respectively; do not return the covered Claims, events, or liability triggers; - return surviving article numbers as strings in the order stated; - return the short jurisdiction name for governing law; - for `miscellaneous.assignment_requires_consent`, report whether Section 10.5's general assignment rule requires the other party's written consent; retain `true` when that general rule has enumerated no-consent exceptions; and - for amendment formality, reproduce the operative source words beginning `in writing` and ending `hereto`. The requested termination fee and aggregate monetary liability cap are `NOT_STATED` when no such amount or formula is stated. The requested arbitration clause is `NOT_FOUND` when the agreement contains no arbitration provision. Do not treat a consequential-damages exclusion as an aggregate liability cap. Never infer a missing legal term from market practice. Every field named in the evidence requirements must cite one exact supplied URL and a short, contiguous supporting quote from that source. Do not modify, normalize, decode, or substitute either official URL. Do not click, type, subscribe, e-file, create an account, log in, send anything, or take any state-changing action. Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and `BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
Reconstructed runner prompt and output contract
BENCHMARK EXECUTION CONTROL (mandatory):
- Use browser tools for the browser work in this task.
- Every browser_start_session call must explicitly pass provider="kernel".
- Start with the supplied reserved ephemeral site_key and pass
clear_cookies=false exactly. The run plan generated this never-before-issued
profile key so the browser starts empty without triggering asynchronous
deletion of another measured attempt's profile.
- Call browser_start_session exactly once. If that start fails, report the
benchmark error and stop instead of retrying with the same or another key.
- Never omit provider and never request Browserbase or automatic fallback.
- Inspect each browser session result. If it identifies any provider other than
Kernel, stop immediately and return a benchmark error instead of continuing.
- When the browser work is complete, call browser_close with neither session_id
nor close_all so the current chat's session is released without an approval.
- Do not take any state-changing action except the task card's explicitly
allowed benchmark-owned Save draft action.
TASK-SPECIFIC BROWSER IDENTITY (mandatory):
- Every browser_start_session call must pass site_key="benchmark-profile-placeholder" exactly. It is an ephemeral profile identifier frozen in this run plan; do not turn it into a URL or Markdown link.
- Every browser_start_session call must pass clear_cookies=false exactly.
- The only permitted tool names for this task are: list_tools, browser_start_session, browser_navigate, browser_snapshot, browser_scroll, browser_open_tab, browser_switch_tab, browser_close_tab, browser_close.
- The measured tool-invocation counts must satisfy: browser_close: exactly 1; browser_scroll: at least 1; browser_snapshot: at least 2; browser_start_session: exactly 1.
- Every browser_start_session call must pass url="https://www.sec.gov/Archives/edgar/data/1412067/000119312514305083/0001193125-14-305083-index.html" exactly. This is the canonical start URL for the task.
- URL-bearing browser calls may use only these exact resource URLs: https://www.sec.gov/Archives/edgar/data/1412067/000119312514305083/0001193125-14-305083-index.html; https://www.sec.gov/Archives/edgar/data/1412067/000119312514305083/d769262dex101.htm. A fragment-only page or anchor selector is allowed on a listed resource; do not alter its scheme, host, path, or query.
- You must visit every listed URL using at least one URL-bearing browser call; opening only a subset fails the measured run.
Review the supplied immutable SEC filing index and Exhibit 10.1 using only the
two supplied official public URLs and the Kernel browser. Return exactly one
record matching the supplied output contract. This is a read-only contract
review benchmark, not legal advice.
Use the filing index for filing metadata and the exhibit for agreement terms.
Trace defined-term pointers to their destination sections instead of treating
the pointer as the full definition. Reconcile the reciprocal indemnities with
the separate joint-indemnification section. Distinguish the material-breach
cure period from the convenience-termination notice period.
Use these deterministic value forms:
- retain the SEC's accession, form, exhibit, filer, and agreement labels;
- format dates as `YYYY-MM-DD`;
- list the two party names in recital order, without addresses or corporate
jurisdiction text;
- format section labels as `Section X` or `Section X.Y`;
- reproduce the capitalized defined-term phrase used as the term-expiration
trigger;
- for `indemnification.oncothyreon_indemnifies` and
`indemnification.celldex_indemnifies`, return only the exact capitalized
defined name of the group indemnified in Sections 8.1 and 8.2, respectively;
do not return the covered Claims, events, or liability triggers;
- return surviving article numbers as strings in the order stated;
- return the short jurisdiction name for governing law;
- for `miscellaneous.assignment_requires_consent`, report whether Section 10.5's
general assignment rule requires the other party's written consent; retain
`true` when that general rule has enumerated no-consent exceptions; and
- for amendment formality, reproduce the operative source words beginning
`in writing` and ending `hereto`.
The requested termination fee and aggregate monetary liability cap are
`NOT_STATED` when no such amount or formula is stated. The requested arbitration
clause is `NOT_FOUND` when the agreement contains no arbitration provision.
Do not treat a consequential-damages exclusion as an aggregate liability cap.
Never infer a missing legal term from market practice.
Every field named in the evidence requirements must cite one exact supplied
URL and a short, contiguous supporting quote from that source. Do not modify,
normalize, decode, or substitute either official URL. Do not click, type,
subscribe, e-file, create an account, log in, send anything, or take any
state-changing action.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
REQUIRED OUTPUT JSON SCHEMA:
{
"$defs": {
"citation": {
"additionalProperties": false,
"properties": {
"quote": {
"minLength": 1,
"type": "string"
},
"url": {
"format": "uri",
"type": "string"
}
},
"required": [
"url",
"quote"
],
"type": "object"
},
"sectionLabel": {
"pattern": "^Section [0-9]+(?:\\.[0-9]+)*$",
"type": "string"
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema",
"items": {
"additionalProperties": false,
"properties": {
"agreement": {
"additionalProperties": false,
"properties": {
"effective_date": {
"format": "date",
"type": "string"
},
"expiration_trigger": {
"minLength": 1,
"type": "string"
},
"parties": {
"items": {
"minLength": 1,
"type": "string"
},
"maxItems": 2,
"minItems": 2,
"type": "array"
},
"title": {
"minLength": 1,
"type": "string"
}
},
"required": [
"title",
"effective_date",
"parties",
"expiration_trigger"
],
"type": "object"
},
"defined_terms": {
"additionalProperties": false,
"properties": {
"claims": {
"additionalProperties": false,
"properties": {
"definition_section": {
"$ref": "#/$defs/sectionLabel"
},
"includes_reasonable_legal_costs": {
"type": "boolean"
}
},
"required": [
"definition_section",
"includes_reasonable_legal_costs"
],
"type": "object"
},
"confidential_information": {
"additionalProperties": false,
"properties": {
"definition_section": {
"$ref": "#/$defs/sectionLabel"
},
"pointer_section": {
"$ref": "#/$defs/sectionLabel"
},
"reasonable_person_override": {
"type": "boolean"
}
},
"required": [
"pointer_section",
"definition_section",
"reasonable_person_override"
],
"type": "object"
}
},
"required": [
"confidential_information",
"claims"
],
"type": "object"
},
"evidence": {
"additionalProperties": false,
"properties": {
"agreement.effective_date": {
"$ref": "#/$defs/citation"
},
"agreement.expiration_trigger": {
"$ref": "#/$defs/citation"
},
"agreement.parties": {
"$ref": "#/$defs/citation"
},
"defined_terms.claims.definition_section": {
"$ref": "#/$defs/citation"
},
"defined_terms.confidential_information.definition_section": {
"$ref": "#/$defs/citation"
},
"defined_terms.confidential_information.reasonable_person_override": {
"$ref": "#/$defs/citation"
},
"filing.accession_number": {
"$ref": "#/$defs/citation"
},
"indemnification.celldex_indemnifies": {
"$ref": "#/$defs/citation"
},
"indemnification.joint_and_several_liability": {
"$ref": "#/$defs/citation"
},
"indemnification.nonsole_injury_cross_reference": {
"$ref": "#/$defs/citation"
},
"indemnification.oncothyreon_indemnifies": {
"$ref": "#/$defs/citation"
},
"miscellaneous.amendment_formality": {
"$ref": "#/$defs/citation"
},
"miscellaneous.assignment_requires_consent": {
"$ref": "#/$defs/citation"
},
"miscellaneous.governing_law": {
"$ref": "#/$defs/citation"
},
"termination.breach_cure_days": {
"$ref": "#/$defs/citation"
},
"termination.convenience_notice_days": {
"$ref": "#/$defs/citation"
},
"termination.surviving_articles": {
"$ref": "#/$defs/citation"
}
},
"required": [
"filing.accession_number",
"agreement.effective_date",
"agreement.parties",
"agreement.expiration_trigger",
"defined_terms.confidential_information.definition_section",
"defined_terms.confidential_information.reasonable_person_override",
"defined_terms.claims.definition_section",
"termination.breach_cure_days",
"termination.convenience_notice_days",
"termination.surviving_articles",
"indemnification.oncothyreon_indemnifies",
"indemnification.celldex_indemnifies",
"indemnification.nonsole_injury_cross_reference",
"indemnification.joint_and_several_liability",
"miscellaneous.governing_law",
"miscellaneous.assignment_requires_consent",
"miscellaneous.amendment_formality"
],
"type": "object"
},
"filing": {
"additionalProperties": false,
"properties": {
"accession_number": {
"pattern": "^[0-9]{10}-[0-9]{2}-[0-9]{6}$",
"type": "string"
},
"exhibit_number": {
"minLength": 1,
"type": "string"
},
"filer": {
"minLength": 1,
"type": "string"
},
"filing_date": {
"format": "date",
"type": "string"
},
"form": {
"minLength": 1,
"type": "string"
}
},
"required": [
"accession_number",
"form",
"filing_date",
"filer",
"exhibit_number"
],
"type": "object"
},
"indemnification": {
"additionalProperties": false,
"properties": {
"aggregate_liability_cap": {
"minLength": 1,
"type": "string"
},
"celldex_indemnifies": {
"description": "Exact capitalized defined name of the group indemnified by Celldex in Section 8.2, not the covered claims or liability triggers.",
"minLength": 1,
"type": "string"
},
"joint_and_several_liability": {
"type": "boolean"
},
"nonsole_injury_cross_reference": {
"$ref": "#/$defs/sectionLabel"
},
"oncothyreon_indemnifies": {
"description": "Exact capitalized defined name of the group indemnified by Oncothyreon in Section 8.1, not the covered claims or liability triggers.",
"minLength": 1,
"type": "string"
}
},
"required": [
"oncothyreon_indemnifies",
"celldex_indemnifies",
"nonsole_injury_cross_reference",
"joint_and_several_liability",
"aggregate_liability_cap"
],
"type": "object"
},
"miscellaneous": {
"additionalProperties": false,
"properties": {
"amendment_formality": {
"minLength": 1,
"type": "string"
},
"arbitration_clause": {
"minLength": 1,
"type": "string"
},
"assignment_requires_consent": {
"description": "Whether Section 10.5's general assignment rule requires the other party's written consent, even when the section provides enumerated no-consent exceptions.",
"type": "boolean"
},
"governing_law": {
"minLength": 1,
"type": "string"
}
},
"required": [
"governing_law",
"assignment_requires_consent",
"amendment_formality",
"arbitration_clause"
],
"type": "object"
},
"official_urls": {
"additionalProperties": false,
"properties": {
"exhibit": {
"format": "uri",
"type": "string"
},
"filing_index": {
"format": "uri",
"type": "string"
}
},
"required": [
"filing_index",
"exhibit"
],
"type": "object"
},
"termination": {
"additionalProperties": false,
"properties": {
"breach_cure_days": {
"minimum": 0,
"type": "integer"
},
"convenience_notice_days": {
"minimum": 0,
"type": "integer"
},
"for_cause_section": {
"$ref": "#/$defs/sectionLabel"
},
"for_convenience_section": {
"$ref": "#/$defs/sectionLabel"
},
"surviving_articles": {
"items": {
"pattern": "^[0-9]+$",
"type": "string"
},
"minItems": 1,
"type": "array",
"uniqueItems": true
},
"termination_fee": {
"minLength": 1,
"type": "string"
}
},
"required": [
"for_cause_section",
"breach_cure_days",
"for_convenience_section",
"convenience_notice_days",
"surviving_articles",
"termination_fee"
],
"type": "object"
}
},
"required": [
"filing",
"agreement",
"defined_terms",
"termination",
"indemnification",
"miscellaneous",
"official_urls",
"evidence"
],
"type": "object"
},
"maxItems": 1,
"minItems": 1,
"type": "array"
}
FROZEN TASK INPUT:
# Frozen task input: Oncothyreon Form 10-Q Exhibit 10.1
Review SEC accession `0001193125-14-305083` and its Phase I Co-Development
Agreement filed as Exhibit 10.1.
Use only these exact official public sources:
- Filing index: https://www.sec.gov/Archives/edgar/data/1412067/000119312514305083/0001193125-14-305083-index.html
- Exhibit 10.1: https://www.sec.gov/Archives/edgar/data/1412067/000119312514305083/d769262dex101.htm
Return exactly these data fields:
- `filing.accession_number`
- `filing.form`
- `filing.filing_date`
- `filing.filer`
- `filing.exhibit_number`
- `agreement.title`
- `agreement.effective_date`
- `agreement.parties`
- `agreement.expiration_trigger`
- `defined_terms.confidential_information.pointer_section`
- `defined_terms.confidential_information.definition_section`
- `defined_terms.confidential_information.reasonable_person_override`
- `defined_terms.claims.definition_section`
- `defined_terms.claims.includes_reasonable_legal_costs`
- `termination.for_cause_section`
- `termination.breach_cure_days`
- `termination.for_convenience_section`
- `termination.convenience_notice_days`
- `termination.surviving_articles`
- `termination.termination_fee` (explicit silence trap)
- `indemnification.oncothyreon_indemnifies`
- `indemnification.celldex_indemnifies`
- `indemnification.nonsole_injury_cross_reference`
- `indemnification.joint_and_several_liability`
- `indemnification.aggregate_liability_cap` (explicit silence trap)
- `miscellaneous.governing_law`
- `miscellaneous.assignment_requires_consent`
- `miscellaneous.amendment_formality`
- `miscellaneous.arbitration_clause` (explicit absence trap)
- `official_urls.filing_index`
- `official_urls.exhibit`
- `evidence`
Evidence is a top-level object keyed by the exact dotted field path. Provide one
`{"url": "...", "quote": "..."}` object for each of these fields:
- `filing.accession_number`
- `agreement.effective_date`
- `agreement.parties`
- `agreement.expiration_trigger`
- `defined_terms.confidential_information.definition_section`
- `defined_terms.confidential_information.reasonable_person_override`
- `defined_terms.claims.definition_section`
- `termination.breach_cure_days`
- `termination.convenience_notice_days`
- `termination.surviving_articles`
- `indemnification.oncothyreon_indemnifies`
- `indemnification.celldex_indemnifies`
- `indemnification.nonsole_injury_cross_reference`
- `indemnification.joint_and_several_liability`
- `miscellaneous.governing_law`
- `miscellaneous.assignment_requires_consent`
- `miscellaneous.amendment_formality`
Use only the two exact URLs above for citations. Quotes must be short contiguous
spans of visible source text; whitespace may be normalized.
Expected output records — evaluator only
[
{
"filing": {
"accession_number": "0001193125-14-305083",
"form": "10-Q",
"filing_date": "2014-08-11",
"filer": "Oncothyreon Inc.",
"exhibit_number": "EX-10.1"
},
"agreement": {
"title": "PHASE I CO-DEVELOPMENT AGREEMENT",
"effective_date": "2014-05-28",
"parties": [
"Celldex Therapeutics, Inc.",
"Oncothyreon Inc."
],
"expiration_trigger": "Completion of the Trial"
},
"defined_terms": {
"confidential_information": {
"pointer_section": "Section 1.8",
"definition_section": "Section 7.1",
"reasonable_person_override": true
},
"claims": {
"definition_section": "Section 1.4",
"includes_reasonable_legal_costs": true
}
},
"termination": {
"for_cause_section": "Section 9.2",
"breach_cure_days": 90,
"for_convenience_section": "Section 9.3",
"convenience_notice_days": 30,
"surviving_articles": [
"4",
"5",
"7",
"8",
"9",
"10"
],
"termination_fee": "NOT_STATED"
},
"indemnification": {
"oncothyreon_indemnifies": "Celldex Indemnitees",
"celldex_indemnifies": "Oncothyreon Indemnitees",
"nonsole_injury_cross_reference": "Section 8.3",
"joint_and_several_liability": true,
"aggregate_liability_cap": "NOT_STATED"
},
"miscellaneous": {
"governing_law": "Delaware",
"assignment_requires_consent": true,
"amendment_formality": "in writing signed by all Parties hereto",
"arbitration_clause": "NOT_FOUND"
},
"official_urls": {
"filing_index": "https://www.sec.gov/Archives/edgar/data/1412067/000119312514305083/0001193125-14-305083-index.html",
"exhibit": "https://www.sec.gov/Archives/edgar/data/1412067/000119312514305083/d769262dex101.htm"
}
}
]Field comparison and evidence rules
{
"record_key": "filing.accession_number",
"success_threshold": "0.90",
"evidence_threshold": "1.00",
"field_rules": [
{
"name": "filing.accession_number",
"kind": "exact",
"critical": true,
"evidence_source": "filing_index"
},
{
"name": "filing.form",
"kind": "exact"
},
{
"name": "filing.filing_date",
"kind": "date"
},
{
"name": "filing.filer",
"kind": "exact"
},
{
"name": "filing.exhibit_number",
"kind": "exact"
},
{
"name": "agreement.title",
"kind": "exact"
},
{
"name": "agreement.effective_date",
"kind": "date",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "agreement.parties",
"kind": "ordered_list",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "agreement.expiration_trigger",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "defined_terms.confidential_information.pointer_section",
"kind": "exact"
},
{
"name": "defined_terms.confidential_information.definition_section",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "defined_terms.confidential_information.reasonable_person_override",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "defined_terms.claims.definition_section",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "defined_terms.claims.includes_reasonable_legal_costs",
"kind": "exact",
"critical": true
},
{
"name": "termination.for_cause_section",
"kind": "exact"
},
{
"name": "termination.breach_cure_days",
"kind": "integer",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "termination.for_convenience_section",
"kind": "exact"
},
{
"name": "termination.convenience_notice_days",
"kind": "integer",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "termination.surviving_articles",
"kind": "ordered_list",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "termination.termination_fee",
"kind": "exact",
"critical": true
},
{
"name": "indemnification.oncothyreon_indemnifies",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "indemnification.celldex_indemnifies",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "indemnification.nonsole_injury_cross_reference",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "indemnification.joint_and_several_liability",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "indemnification.aggregate_liability_cap",
"kind": "exact",
"critical": true
},
{
"name": "miscellaneous.governing_law",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "miscellaneous.assignment_requires_consent",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "miscellaneous.amendment_formality",
"kind": "exact",
"critical": true,
"evidence_source": "agreement"
},
{
"name": "miscellaneous.arbitration_clause",
"kind": "exact",
"critical": true
},
{
"name": "official_urls.filing_index",
"kind": "exact"
},
{
"name": "official_urls.exhibit",
"kind": "exact"
}
],
"evidence_sources": {
"filing_index": {
"url": "https://www.sec.gov/Archives/edgar/data/1412067/000119312514305083/0001193125-14-305083-index.html",
"path": "fixtures/filing-index.html",
"format": "html",
"require_quote": true,
"normalize_quote_whitespace": true
},
"agreement": {
"url": "https://www.sec.gov/Archives/edgar/data/1412067/000119312514305083/d769262dex101.htm",
"path": "fixtures/celldex-oncothyreon-phase-i-agreement.html",
"format": "html",
"require_quote": true,
"normalize_quote_whitespace": true
}
}
}Court docket
Groff v. DeJoy Supreme Court docket chronology. Pack 4; historical execution pack.
Executed by: Claude Fable 5.1, Claude Opus 5, Claude Sonnet 5, GPT-5.6 Sol, Gemini 3.8 Flash, Grok 4.6.
Time limit: 15 minutes.
Supplied source URLs
Task prompt template
Research the supplied concluded Supreme Court docket using only the supplied
official public URLs and the Kernel browser. Return exactly one record matching
the supplied output contract. Treat "last three pre-disposition events" as the
last three Proceedings and Orders rows strictly before the decision row; do not
make a subjective materiality judgment.
For each selected event, reproduce the complete visible row description with
whitespace normalized; do not shorten the argument row to only `Argued.`.
For `last_three_pre_disposition_events` evidence, return three distinct
citations in event order, one exact docket quote for each selected row.
Use these deterministic value forms: retain `Petitioner` in the one-line case
caption; strip surrounding parentheses from the lower-court case number; format
the opinion author as `Justice {surname}`; format a concurrence as `Justice
{author} (joined by Justice {joiner})`; and reproduce the text after the
syllabus `Held:` label verbatim, omitting the `Held:` label itself, with
whitespace and line-break hyphenation normalized. Return only the holding
sentence: stop after `business.` and omit the terminal syllabus page-range
citation `Pp. 4–21.`.
Normalize `disposition.result` to exactly one of the following values based on
the disposition stated on the docket: `AFFIRMED`, `REVERSED`,
`REVERSED_AND_REMANDED`, `VACATED`, `VACATED_AND_REMANDED`, `DISMISSED`, or
`OTHER`. Use `VACATED_AND_REMANDED` when the docket states that the lower-court
judgment was vacated and the case was remanded for further proceedings. Use
`OTHER` only when none of the other listed dispositions accurately applies.
Every field named in the evidence requirements must cite a short supporting
quote from the designated supplied URL. Use the docket URL for
`case.docket_number`, `dates.decided`, `disposition.result`,
`opinion.unanimous`, and each `last_three_pre_disposition_events` citation. Use
the opinion URL for `opinion.author` and `holding`. Use the literal
`NOT_STATED` when the source is silent and `NOT_FOUND` when the requested item
is absent; never infer a legal fact. Do not subscribe, e-file, create an
account, follow a filing workflow, or take any state-changing action.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.Reconstructed runner prompt and output contract
BENCHMARK EXECUTION CONTROL (mandatory):
- Use browser tools for the browser work in this task.
- Every browser_start_session call must explicitly pass provider="kernel".
- Start with the supplied reserved ephemeral site_key and pass
clear_cookies=false exactly. The run plan generated this never-before-issued
profile key so the browser starts empty without triggering asynchronous
deletion of another measured attempt's profile.
- Call browser_start_session exactly once. If that start fails, report the
benchmark error and stop instead of retrying with the same or another key.
- Never omit provider and never request Browserbase or automatic fallback.
- Inspect each browser session result. If it identifies any provider other than
Kernel, stop immediately and return a benchmark error instead of continuing.
- When the browser work is complete, call browser_close with neither session_id
nor close_all so the current chat's session is released without an approval.
- Do not take any state-changing action except the task card's explicitly
allowed benchmark-owned Save draft action.
TASK-SPECIFIC BROWSER IDENTITY (mandatory):
- Every browser_start_session call must pass site_key="benchmark-profile-placeholder" exactly. It is an ephemeral profile identifier frozen in this run plan; do not turn it into a URL or Markdown link.
- Every browser_start_session call must pass clear_cookies=false exactly.
- The only permitted tool names for this task are: list_tools, browser_start_session, browser_navigate, browser_snapshot, browser_scroll, browser_open_tab, browser_switch_tab, browser_close_tab, browser_close.
- Every browser_start_session call must pass url="https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html" exactly. This is the canonical start URL for the task.
- URL-bearing browser calls may use only these exact resource URLs: https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html; https://www.supremecourt.gov/opinions/22pdf/22-174_k536.pdf. A fragment-only page or anchor selector is allowed on a listed resource; do not alter its scheme, host, path, or query.
- You must visit every listed URL using at least one URL-bearing browser call; opening only a subset fails the measured run.
Research the supplied concluded Supreme Court docket using only the supplied
official public URLs and the Kernel browser. Return exactly one record matching
the supplied output contract. Treat "last three pre-disposition events" as the
last three Proceedings and Orders rows strictly before the decision row; do not
make a subjective materiality judgment.
For each selected event, reproduce the complete visible row description with
whitespace normalized; do not shorten the argument row to only `Argued.`.
For `last_three_pre_disposition_events` evidence, return three distinct
citations in event order, one exact docket quote for each selected row.
Use these deterministic value forms: retain `Petitioner` in the one-line case
caption; strip surrounding parentheses from the lower-court case number; format
the opinion author as `Justice {surname}`; format a concurrence as `Justice
{author} (joined by Justice {joiner})`; and reproduce the text after the
syllabus `Held:` label verbatim, omitting the `Held:` label itself, with
whitespace and line-break hyphenation normalized. Return only the holding
sentence: stop after `business.` and omit the terminal syllabus page-range
citation `Pp. 4–21.`.
Normalize `disposition.result` to exactly one of the following values based on
the disposition stated on the docket: `AFFIRMED`, `REVERSED`,
`REVERSED_AND_REMANDED`, `VACATED`, `VACATED_AND_REMANDED`, `DISMISSED`, or
`OTHER`. Use `VACATED_AND_REMANDED` when the docket states that the lower-court
judgment was vacated and the case was remanded for further proceedings. Use
`OTHER` only when none of the other listed dispositions accurately applies.
Every field named in the evidence requirements must cite a short supporting
quote from the designated supplied URL. Use the docket URL for
`case.docket_number`, `dates.decided`, `disposition.result`,
`opinion.unanimous`, and each `last_three_pre_disposition_events` citation. Use
the opinion URL for `opinion.author` and `holding`. Use the literal
`NOT_STATED` when the source is silent and `NOT_FOUND` when the requested item
is absent; never infer a legal fact. Do not subscribe, e-file, create an
account, follow a filing workflow, or take any state-changing action.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
REQUIRED OUTPUT JSON SCHEMA:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"items": {
"additionalProperties": false,
"properties": {
"case": {
"additionalProperties": false,
"properties": {
"docket_number": {
"type": "string"
},
"name": {
"type": "string"
}
},
"required": [
"name",
"docket_number"
],
"type": "object"
},
"dates": {
"additionalProperties": false,
"properties": {
"argued": {
"format": "date",
"type": "string"
},
"decided": {
"format": "date",
"type": "string"
},
"docketed": {
"format": "date",
"type": "string"
}
},
"required": [
"docketed",
"argued",
"decided"
],
"type": "object"
},
"disposition": {
"additionalProperties": false,
"properties": {
"judgment_issued_date": {
"format": "date",
"type": "string"
},
"result": {
"enum": [
"AFFIRMED",
"REVERSED",
"REVERSED_AND_REMANDED",
"VACATED",
"VACATED_AND_REMANDED",
"DISMISSED",
"OTHER"
],
"type": "string"
}
},
"required": [
"result",
"judgment_issued_date"
],
"type": "object"
},
"evidence": {
"additionalProperties": {
"additionalProperties": false,
"properties": {
"quote": {
"minLength": 1,
"type": "string"
},
"url": {
"format": "uri",
"type": "string"
}
},
"required": [
"url",
"quote"
],
"type": "object"
},
"properties": {
"last_three_pre_disposition_events": {
"items": {
"additionalProperties": false,
"properties": {
"quote": {
"minLength": 1,
"type": "string"
},
"url": {
"format": "uri",
"type": "string"
}
},
"required": [
"url",
"quote"
],
"type": "object"
},
"maxItems": 3,
"minItems": 3,
"type": "array"
}
},
"required": [
"case.docket_number",
"dates.decided",
"disposition.result",
"opinion.author",
"opinion.unanimous",
"last_three_pre_disposition_events",
"holding"
],
"type": "object"
},
"holding": {
"description": "The syllabus holding sentence after Held:, without the Held: label or terminal Pp. page-range citation.",
"type": "string"
},
"last_three_pre_disposition_events": {
"items": {
"additionalProperties": false,
"properties": {
"date": {
"format": "date",
"type": "string"
},
"description": {
"type": "string"
}
},
"required": [
"date",
"description"
],
"type": "object"
},
"maxItems": 3,
"minItems": 3,
"type": "array"
},
"lower_court": {
"additionalProperties": false,
"properties": {
"case_number": {
"type": "string"
},
"decision_date": {
"format": "date",
"type": "string"
},
"name": {
"type": "string"
},
"panel_vote": {
"minLength": 1,
"type": "string"
}
},
"required": [
"name",
"case_number",
"decision_date",
"panel_vote"
],
"type": "object"
},
"official_urls": {
"additionalProperties": false,
"properties": {
"docket": {
"format": "uri",
"type": "string"
},
"opinion": {
"format": "uri",
"type": "string"
}
},
"required": [
"docket",
"opinion"
],
"type": "object"
},
"opinion": {
"additionalProperties": false,
"properties": {
"author": {
"type": "string"
},
"concurrences": {
"items": {
"type": "string"
},
"type": "array"
},
"dissent_author": {
"minLength": 1,
"type": "string"
},
"majority_count": {
"oneOf": [
{
"minimum": 0,
"type": "integer"
},
{
"minLength": 1,
"type": "string"
}
]
},
"unanimous": {
"type": "boolean"
}
},
"required": [
"author",
"unanimous",
"majority_count",
"concurrences",
"dissent_author"
],
"type": "object"
}
},
"required": [
"case",
"lower_court",
"dates",
"disposition",
"opinion",
"last_three_pre_disposition_events",
"holding",
"official_urls",
"evidence"
],
"type": "object"
},
"maxItems": 1,
"minItems": 1,
"type": "array"
}
FROZEN TASK INPUT:
# Frozen task input: Groff v. DeJoy
Research Supreme Court docket `22-174`, *Gerald E. Groff v. Louis DeJoy,
Postmaster General*.
Use only these official public sources:
- Docket: https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html
- Slip opinion: https://www.supremecourt.gov/opinions/22pdf/22-174_k536.pdf
Return exactly these data fields:
- `case.name`
- `case.docket_number`
- `lower_court.name`
- `lower_court.case_number`
- `lower_court.decision_date`
- `lower_court.panel_vote` (an explicit abstention trap)
- `dates.docketed`
- `dates.argued`
- `dates.decided`
- `disposition.result`
- `disposition.judgment_issued_date`
- `opinion.author`
- `opinion.unanimous`
- `opinion.majority_count`
- `opinion.concurrences`
- `opinion.dissent_author` (an explicit absence trap)
- `last_three_pre_disposition_events`
- `holding`
- `official_urls.docket`
- `official_urls.opinion`
- `evidence`
Each event is an object with `date` and `description`. Evidence is a top-level
object keyed by the exact dotted field path. Provide evidence for:
- `case.docket_number`
- `dates.decided`
- `disposition.result`
- `opinion.author`
- `opinion.unanimous`
- `last_three_pre_disposition_events`
- `holding`
For every field except `last_three_pre_disposition_events`, the evidence value
must be one `{"url": "...", "quote": "..."}` object. The evidence value for
`last_three_pre_disposition_events` must be an array of exactly three such
objects, one citation for each returned event in the same order. Use only the
two exact URLs above.
Expected output records — evaluator only
[
{
"case": {
"name": "Gerald E. Groff, Petitioner v. Louis DeJoy, Postmaster General",
"docket_number": "22-174"
},
"lower_court": {
"name": "United States Court of Appeals for the Third Circuit",
"case_number": "21-1900",
"decision_date": "2022-05-25",
"panel_vote": "NOT_STATED"
},
"dates": {
"docketed": "2022-08-25",
"argued": "2023-04-18",
"decided": "2023-06-29"
},
"disposition": {
"result": "VACATED_AND_REMANDED",
"judgment_issued_date": "2023-07-31"
},
"opinion": {
"author": "Justice Alito",
"unanimous": true,
"majority_count": "NOT_STATED",
"concurrences": [
"Justice Sotomayor (joined by Justice Jackson)"
],
"dissent_author": "NOT_FOUND"
},
"last_three_pre_disposition_events": [
{
"date": "2023-04-07",
"description": "Letter filed. (Distributed)"
},
{
"date": "2023-04-07",
"description": "Reply of petitioner Gerald E. Groff filed. (Distributed)"
},
{
"date": "2023-04-18",
"description": "Argued. For petitioner: Aaron Streett, Houston, Tex. For respondent: Elizabeth B. Prelogar, Solicitor General, Department of Justice, Washington, D. C."
}
],
"holding": "Title VII requires an employer that denies a religious accommodation to show that the burden of granting an accommodation would result in substantial increased costs in relation to the conduct of its particular business.",
"official_urls": {
"docket": "https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html",
"opinion": "https://www.supremecourt.gov/opinions/22pdf/22-174_k536.pdf"
}
}
]Field comparison and evidence rules
{
"record_key": "case.docket_number",
"success_threshold": "0.90",
"evidence_threshold": "1.00",
"field_rules": [
{
"name": "case.name",
"kind": "exact"
},
{
"name": "case.docket_number",
"kind": "exact",
"critical": true,
"evidence_source": "docket"
},
{
"name": "lower_court.name",
"kind": "exact"
},
{
"name": "lower_court.case_number",
"kind": "exact"
},
{
"name": "lower_court.decision_date",
"kind": "date"
},
{
"name": "lower_court.panel_vote",
"kind": "exact"
},
{
"name": "dates.docketed",
"kind": "date"
},
{
"name": "dates.argued",
"kind": "date"
},
{
"name": "dates.decided",
"kind": "date",
"critical": true,
"evidence_source": "docket"
},
{
"name": "disposition.result",
"kind": "enum",
"enum_values": [
"AFFIRMED",
"REVERSED",
"REVERSED_AND_REMANDED",
"VACATED",
"VACATED_AND_REMANDED",
"DISMISSED",
"OTHER"
],
"critical": true,
"evidence_source": "docket"
},
{
"name": "disposition.judgment_issued_date",
"kind": "date"
},
{
"name": "opinion.author",
"kind": "exact",
"critical": true,
"evidence_source": "opinion"
},
{
"name": "opinion.unanimous",
"kind": "exact",
"evidence_source": "docket"
},
{
"name": "opinion.majority_count",
"kind": "exact"
},
{
"name": "opinion.concurrences",
"kind": "ordered_list"
},
{
"name": "opinion.dissent_author",
"kind": "exact"
},
{
"name": "last_three_pre_disposition_events",
"kind": "ordered_list",
"evidence_source": "docket_events"
},
{
"name": "holding",
"kind": "exact",
"critical": true,
"evidence_source": "opinion"
},
{
"name": "official_urls.docket",
"kind": "exact"
},
{
"name": "official_urls.opinion",
"kind": "exact"
}
],
"evidence_sources": {
"docket": {
"url": "https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html",
"path": "fixtures/groff-docket.html",
"format": "html",
"require_quote": true,
"normalize_quote_whitespace": true
},
"docket_events": {
"url": "https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html",
"path": "fixtures/groff-docket.html",
"format": "html",
"min_citations": 3,
"require_quote": true,
"normalize_quote_whitespace": true
},
"opinion": {
"url": "https://www.supremecourt.gov/opinions/22pdf/22-174_k536.pdf",
"path": "fixtures/groff-opinion-evidence.txt",
"derived_from": "fixtures/groff-opinion.pdf",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true,
"normalize_linebreak_hyphenation": true
}
}
}Court docket
Groff v. DeJoy Supreme Court docket chronology. Pack 5; reviewed scoring pack.
Executed by: GPT-6 Astra, Kimi K3.
Time limit: 15 minutes.
Supplied source URLs
Task prompt template
Research the supplied concluded Supreme Court docket using only the supplied
official public URLs and the Kernel browser. Return exactly one record matching
the supplied output contract. Treat "last three pre-disposition events" as the
last three Proceedings and Orders rows strictly before the decision row; do not
make a subjective materiality judgment.
For each selected event, reproduce the complete visible row description with
whitespace normalized; do not shorten the argument row to only `Argued.`.
For `last_three_pre_disposition_events` evidence, return three distinct
citations in event order, one exact docket quote for each selected row.
Use these deterministic value forms: retain `Petitioner` in the one-line case
caption; strip surrounding parentheses from the lower-court case number; format
the opinion author as `Justice {surname}`; format a concurrence as `Justice
{author} (joined by Justice {joiner})`; and reproduce the text after the
syllabus `Held:` label verbatim, omitting the `Held:` label itself, with
whitespace and line-break hyphenation normalized. Return only the holding
sentence: stop after `business.` and omit the terminal syllabus page-range
citation `Pp. 4–21.`.
Normalize `disposition.result` to exactly one of the following values based on
the disposition stated on the docket: `AFFIRMED`, `REVERSED`,
`REVERSED_AND_REMANDED`, `VACATED`, `VACATED_AND_REMANDED`, `DISMISSED`, or
`OTHER`. Use `VACATED_AND_REMANDED` when the docket states that the lower-court
judgment was vacated and the case was remanded for further proceedings. Use
`OTHER` only when none of the other listed dispositions accurately applies.
Every field named in the evidence requirements must cite a short supporting
quote from the designated supplied URL. Use the docket URL for
`case.docket_number`, `dates.decided`, `disposition.result`,
`opinion.unanimous`, and each `last_three_pre_disposition_events` citation. Use
the opinion URL for `opinion.author` and `holding`. Use the literal
`NOT_STATED` when the source is silent and `NOT_FOUND` when the requested item
is absent; never infer a legal fact. Do not subscribe, e-file, create an
account, follow a filing workflow, or take any state-changing action.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.Reconstructed runner prompt and output contract
BENCHMARK EXECUTION CONTROL (mandatory):
- Use browser tools for the browser work in this task.
- Every browser_start_session call must explicitly pass provider="kernel".
- Start with the supplied reserved ephemeral site_key and pass
clear_cookies=false exactly. The run plan generated this never-before-issued
profile key so the browser starts empty without triggering asynchronous
deletion of another measured attempt's profile.
- Call browser_start_session exactly once. If that start fails, report the
benchmark error and stop instead of retrying with the same or another key.
- Never omit provider and never request Browserbase or automatic fallback.
- Inspect each browser session result. If it identifies any provider other than
Kernel, stop immediately and return a benchmark error instead of continuing.
- When the browser work is complete, call browser_close with neither session_id
nor close_all so the current chat's session is released without an approval.
- Do not take any state-changing action except the task card's explicitly
allowed benchmark-owned Save draft action.
TASK-SPECIFIC BROWSER IDENTITY (mandatory):
- Every browser_start_session call must pass site_key="benchmark-profile-placeholder" exactly. It is an ephemeral profile identifier frozen in this run plan; do not turn it into a URL or Markdown link.
- Every browser_start_session call must pass clear_cookies=false exactly.
- The only permitted tool names for this task are: list_tools, browser_start_session, browser_navigate, browser_snapshot, browser_scroll, browser_open_tab, browser_switch_tab, browser_close_tab, browser_close.
- Every browser_start_session call must pass url="https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html" exactly. This is the canonical start URL for the task.
- URL-bearing browser calls may use only these exact resource URLs: https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html; https://www.supremecourt.gov/opinions/22pdf/22-174_k536.pdf. A fragment-only page or anchor selector is allowed on a listed resource; do not alter its scheme, host, path, or query.
- You must visit every listed URL using at least one URL-bearing browser call; opening only a subset fails the measured run.
Research the supplied concluded Supreme Court docket using only the supplied
official public URLs and the Kernel browser. Return exactly one record matching
the supplied output contract. Treat "last three pre-disposition events" as the
last three Proceedings and Orders rows strictly before the decision row; do not
make a subjective materiality judgment.
For each selected event, reproduce the complete visible row description with
whitespace normalized; do not shorten the argument row to only `Argued.`.
For `last_three_pre_disposition_events` evidence, return three distinct
citations in event order, one exact docket quote for each selected row.
Use these deterministic value forms: retain `Petitioner` in the one-line case
caption; strip surrounding parentheses from the lower-court case number; format
the opinion author as `Justice {surname}`; format a concurrence as `Justice
{author} (joined by Justice {joiner})`; and reproduce the text after the
syllabus `Held:` label verbatim, omitting the `Held:` label itself, with
whitespace and line-break hyphenation normalized. Return only the holding
sentence: stop after `business.` and omit the terminal syllabus page-range
citation `Pp. 4–21.`.
Normalize `disposition.result` to exactly one of the following values based on
the disposition stated on the docket: `AFFIRMED`, `REVERSED`,
`REVERSED_AND_REMANDED`, `VACATED`, `VACATED_AND_REMANDED`, `DISMISSED`, or
`OTHER`. Use `VACATED_AND_REMANDED` when the docket states that the lower-court
judgment was vacated and the case was remanded for further proceedings. Use
`OTHER` only when none of the other listed dispositions accurately applies.
Every field named in the evidence requirements must cite a short supporting
quote from the designated supplied URL. Use the docket URL for
`case.docket_number`, `dates.decided`, `disposition.result`,
`opinion.unanimous`, and each `last_three_pre_disposition_events` citation. Use
the opinion URL for `opinion.author` and `holding`. Use the literal
`NOT_STATED` when the source is silent and `NOT_FOUND` when the requested item
is absent; never infer a legal fact. Do not subscribe, e-file, create an
account, follow a filing workflow, or take any state-changing action.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
REQUIRED OUTPUT JSON SCHEMA:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"items": {
"additionalProperties": false,
"properties": {
"case": {
"additionalProperties": false,
"properties": {
"docket_number": {
"type": "string"
},
"name": {
"type": "string"
}
},
"required": [
"name",
"docket_number"
],
"type": "object"
},
"dates": {
"additionalProperties": false,
"properties": {
"argued": {
"format": "date",
"type": "string"
},
"decided": {
"format": "date",
"type": "string"
},
"docketed": {
"format": "date",
"type": "string"
}
},
"required": [
"docketed",
"argued",
"decided"
],
"type": "object"
},
"disposition": {
"additionalProperties": false,
"properties": {
"judgment_issued_date": {
"format": "date",
"type": "string"
},
"result": {
"enum": [
"AFFIRMED",
"REVERSED",
"REVERSED_AND_REMANDED",
"VACATED",
"VACATED_AND_REMANDED",
"DISMISSED",
"OTHER"
],
"type": "string"
}
},
"required": [
"result",
"judgment_issued_date"
],
"type": "object"
},
"evidence": {
"additionalProperties": {
"additionalProperties": false,
"properties": {
"quote": {
"minLength": 1,
"type": "string"
},
"url": {
"format": "uri",
"type": "string"
}
},
"required": [
"url",
"quote"
],
"type": "object"
},
"properties": {
"last_three_pre_disposition_events": {
"items": {
"additionalProperties": false,
"properties": {
"quote": {
"minLength": 1,
"type": "string"
},
"url": {
"format": "uri",
"type": "string"
}
},
"required": [
"url",
"quote"
],
"type": "object"
},
"maxItems": 3,
"minItems": 3,
"type": "array"
}
},
"required": [
"case.docket_number",
"dates.decided",
"disposition.result",
"opinion.author",
"opinion.unanimous",
"last_three_pre_disposition_events",
"holding"
],
"type": "object"
},
"holding": {
"description": "The syllabus holding sentence after Held:, without the Held: label or terminal Pp. page-range citation.",
"type": "string"
},
"last_three_pre_disposition_events": {
"items": {
"additionalProperties": false,
"properties": {
"date": {
"format": "date",
"type": "string"
},
"description": {
"type": "string"
}
},
"required": [
"date",
"description"
],
"type": "object"
},
"maxItems": 3,
"minItems": 3,
"type": "array"
},
"lower_court": {
"additionalProperties": false,
"properties": {
"case_number": {
"type": "string"
},
"decision_date": {
"format": "date",
"type": "string"
},
"name": {
"type": "string"
},
"panel_vote": {
"minLength": 1,
"type": "string"
}
},
"required": [
"name",
"case_number",
"decision_date",
"panel_vote"
],
"type": "object"
},
"official_urls": {
"additionalProperties": false,
"properties": {
"docket": {
"format": "uri",
"type": "string"
},
"opinion": {
"format": "uri",
"type": "string"
}
},
"required": [
"docket",
"opinion"
],
"type": "object"
},
"opinion": {
"additionalProperties": false,
"properties": {
"author": {
"type": "string"
},
"concurrences": {
"items": {
"type": "string"
},
"type": "array"
},
"dissent_author": {
"minLength": 1,
"type": "string"
},
"majority_count": {
"oneOf": [
{
"minimum": 0,
"type": "integer"
},
{
"minLength": 1,
"type": "string"
}
]
},
"unanimous": {
"type": "boolean"
}
},
"required": [
"author",
"unanimous",
"majority_count",
"concurrences",
"dissent_author"
],
"type": "object"
}
},
"required": [
"case",
"lower_court",
"dates",
"disposition",
"opinion",
"last_three_pre_disposition_events",
"holding",
"official_urls",
"evidence"
],
"type": "object"
},
"maxItems": 1,
"minItems": 1,
"type": "array"
}
FROZEN TASK INPUT:
# Frozen task input: Groff v. DeJoy
Research Supreme Court docket `22-174`, *Gerald E. Groff v. Louis DeJoy,
Postmaster General*.
Use only these official public sources:
- Docket: https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html
- Slip opinion: https://www.supremecourt.gov/opinions/22pdf/22-174_k536.pdf
Return exactly these data fields:
- `case.name`
- `case.docket_number`
- `lower_court.name`
- `lower_court.case_number`
- `lower_court.decision_date`
- `lower_court.panel_vote` (an explicit abstention trap)
- `dates.docketed`
- `dates.argued`
- `dates.decided`
- `disposition.result`
- `disposition.judgment_issued_date`
- `opinion.author`
- `opinion.unanimous`
- `opinion.majority_count`
- `opinion.concurrences`
- `opinion.dissent_author` (an explicit absence trap)
- `last_three_pre_disposition_events`
- `holding`
- `official_urls.docket`
- `official_urls.opinion`
- `evidence`
Each event is an object with `date` and `description`. Evidence is a top-level
object keyed by the exact dotted field path. Provide evidence for:
- `case.docket_number`
- `dates.decided`
- `disposition.result`
- `opinion.author`
- `opinion.unanimous`
- `last_three_pre_disposition_events`
- `holding`
For every field except `last_three_pre_disposition_events`, the evidence value
must be one `{"url": "...", "quote": "..."}` object. The evidence value for
`last_three_pre_disposition_events` must be an array of exactly three such
objects, one citation for each returned event in the same order. Use only the
two exact URLs above.
Expected output records — evaluator only
[
{
"case": {
"name": "Gerald E. Groff, Petitioner v. Louis DeJoy, Postmaster General",
"docket_number": "22-174"
},
"lower_court": {
"name": "United States Court of Appeals for the Third Circuit",
"case_number": "21-1900",
"decision_date": "2022-05-25",
"panel_vote": "NOT_STATED"
},
"dates": {
"docketed": "2022-08-25",
"argued": "2023-04-18",
"decided": "2023-06-29"
},
"disposition": {
"result": "VACATED_AND_REMANDED",
"judgment_issued_date": "2023-07-31"
},
"opinion": {
"author": "Justice Alito",
"unanimous": true,
"majority_count": "NOT_STATED",
"concurrences": [
"Justice Sotomayor (joined by Justice Jackson)"
],
"dissent_author": "NOT_FOUND"
},
"last_three_pre_disposition_events": [
{
"date": "2023-04-07",
"description": "Letter filed. (Distributed)"
},
{
"date": "2023-04-07",
"description": "Reply of petitioner Gerald E. Groff filed. (Distributed)"
},
{
"date": "2023-04-18",
"description": "Argued. For petitioner: Aaron Streett, Houston, Tex. For respondent: Elizabeth B. Prelogar, Solicitor General, Department of Justice, Washington, D. C."
}
],
"holding": "Title VII requires an employer that denies a religious accommodation to show that the burden of granting an accommodation would result in substantial increased costs in relation to the conduct of its particular business.",
"official_urls": {
"docket": "https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html",
"opinion": "https://www.supremecourt.gov/opinions/22pdf/22-174_k536.pdf"
}
}
]Field comparison and evidence rules
{
"record_key": "case.docket_number",
"success_threshold": "0.90",
"evidence_threshold": "1.00",
"field_rules": [
{
"name": "case.name",
"kind": "exact"
},
{
"name": "case.docket_number",
"kind": "exact",
"critical": true,
"evidence_source": "docket"
},
{
"name": "lower_court.name",
"kind": "exact"
},
{
"name": "lower_court.case_number",
"kind": "exact"
},
{
"name": "lower_court.decision_date",
"kind": "date"
},
{
"name": "lower_court.panel_vote",
"kind": "exact"
},
{
"name": "dates.docketed",
"kind": "date"
},
{
"name": "dates.argued",
"kind": "date"
},
{
"name": "dates.decided",
"kind": "date",
"critical": true,
"evidence_source": "docket"
},
{
"name": "disposition.result",
"kind": "enum",
"enum_values": [
"AFFIRMED",
"REVERSED",
"REVERSED_AND_REMANDED",
"VACATED",
"VACATED_AND_REMANDED",
"DISMISSED",
"OTHER"
],
"critical": true,
"evidence_source": "docket"
},
{
"name": "disposition.judgment_issued_date",
"kind": "date"
},
{
"name": "opinion.author",
"kind": "exact",
"critical": true,
"evidence_source": "opinion"
},
{
"name": "opinion.unanimous",
"kind": "exact",
"evidence_source": "docket"
},
{
"name": "opinion.majority_count",
"kind": "exact"
},
{
"name": "opinion.concurrences",
"kind": "ordered_list"
},
{
"name": "opinion.dissent_author",
"kind": "exact"
},
{
"name": "last_three_pre_disposition_events",
"kind": "ordered_list",
"evidence_source": "docket_events"
},
{
"name": "holding",
"kind": "exact",
"critical": true,
"evidence_source": "opinion"
},
{
"name": "official_urls.docket",
"kind": "exact"
},
{
"name": "official_urls.opinion",
"kind": "exact"
}
],
"evidence_sources": {
"docket": {
"url": "https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html",
"path": "fixtures/groff-docket.html",
"format": "html",
"require_quote": true,
"normalize_quote_whitespace": true
},
"docket_events": {
"url": "https://www.supremecourt.gov/docket/docketfiles/html/public/22-174.html",
"path": "fixtures/groff-docket.html",
"format": "html",
"min_citations": 3,
"require_quote": true,
"normalize_quote_whitespace": true
},
"opinion": {
"url": "https://www.supremecourt.gov/opinions/22pdf/22-174_k536.pdf",
"path": "fixtures/groff-opinion-evidence.txt",
"derived_from": "fixtures/groff-opinion.pdf",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true,
"normalize_linebreak_hyphenation": true
}
}
}eCFR regulation
ADA title II web-accessibility rule codification trace. Pack 2; reviewed scoring pack.
Executed by: Claude Fable 5.1, Claude Opus 5, Claude Sonnet 5, GPT-5.6 Sol, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.6, Kimi K3.
Time limit: 15 minutes.
Supplied source URLs
Task prompt template
Research the supplied historical eCFR record using only the three supplied
official `www.ecfr.gov` URLs and the Kernel browser. Trace the Federal Register
source note for 28 CFR part 35, subpart H, through the point-in-time change from
June 23 to June 24, 2024. Return exactly one record matching the supplied output
contract.
For `authority.subpart_source_citation`, return only the Federal Register
volume-and-page citation in the form `{volume} FR {page}`; do not return the
surrounding source-note text, AG order, date, `Source:` label, or trailing
`unless otherwise noted`. For `authority.ag_order_number`, retain the complete
visible `AG Order No. {number}` label rather than returning only its number.
Treat the earliest substantive `35.200` content version as the operative
amendment/effective date for this trace. Do not substitute the API metadata's
later `latest_amendment_date`. Reproduce the seven complete post-change section
headings in regulatory order, including section symbols/numbers, final
punctuation, and the reserved range. Reproduce the five
exceptions in paragraph order using their italicized headings without the final
period. Use the canonical covered-content labels `web content` and `mobile apps`,
and the conformance labels `Level A` and `Level AA` in that order.
For `codification.before_subpart_heading` and
`codification.after_subpart_heading`, return only the complete heading beginning
with `Subpart H—`; exclude adjacent cross-reference and source-note text.
Use `NOT_STATED` when a requested metadata item or requirement is not stated in
the supplied sources. Use `NOT_FOUND` when the requested historical codified
item is absent from the supplied point-in-time view. Do not infer a RIN,
Federal Register document number, testing frequency, or pre-effective section
list from outside knowledge.
For every field named in the evidence requirements, cite the exact supplied URL
and a short exact quote from that source. The evidence for section headings,
covered content, and exceptions must include one distinct citation per returned
item in the same order. Do not search the broader web, follow external links,
sign in, subscribe, submit a comment, or take any state-changing action.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.Reconstructed runner prompt and output contract
BENCHMARK EXECUTION CONTROL (mandatory):
- Use browser tools for the browser work in this task.
- Every browser_start_session call must explicitly pass provider="kernel".
- Start with the supplied reserved ephemeral site_key and pass
clear_cookies=false exactly. The run plan generated this never-before-issued
profile key so the browser starts empty without triggering asynchronous
deletion of another measured attempt's profile.
- Call browser_start_session exactly once. If that start fails, report the
benchmark error and stop instead of retrying with the same or another key.
- Never omit provider and never request Browserbase or automatic fallback.
- Inspect each browser session result. If it identifies any provider other than
Kernel, stop immediately and return a benchmark error instead of continuing.
- When the browser work is complete, call browser_close with neither session_id
nor close_all so the current chat's session is released without an approval.
- Do not take any state-changing action except the task card's explicitly
allowed benchmark-owned Save draft action.
TASK-SPECIFIC BROWSER IDENTITY (mandatory):
- Every browser_start_session call must pass site_key="benchmark-profile-placeholder" exactly. It is an ephemeral profile identifier frozen in this run plan; do not turn it into a URL or Markdown link.
- Every browser_start_session call must pass clear_cookies=false exactly.
- The only permitted tool names for this task are: list_tools, browser_start_session, browser_navigate, browser_snapshot, browser_scroll, browser_open_tab, browser_switch_tab, browser_close_tab, browser_close.
- The measured tool-invocation counts must satisfy: browser_close: exactly 1; browser_snapshot: at least 3; browser_start_session: exactly 1.
- Every browser_start_session call must pass url="https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-23/title-28?chapter=I&part=35&subpart=H" exactly. This is the canonical start URL for the task.
- URL-bearing browser calls may use only these exact resource URLs: https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-23/title-28?chapter=I&part=35&subpart=H; https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-24/title-28?chapter=I&part=35&subpart=H; https://www.ecfr.gov/api/versioner/v1/versions/title-28.json?part=35§ion=35.200. A fragment-only page or anchor selector is allowed on a listed resource; do not alter its scheme, host, path, or query.
- You must visit every listed URL using at least one URL-bearing browser call; opening only a subset fails the measured run.
Research the supplied historical eCFR record using only the three supplied
official `www.ecfr.gov` URLs and the Kernel browser. Trace the Federal Register
source note for 28 CFR part 35, subpart H, through the point-in-time change from
June 23 to June 24, 2024. Return exactly one record matching the supplied output
contract.
For `authority.subpart_source_citation`, return only the Federal Register
volume-and-page citation in the form `{volume} FR {page}`; do not return the
surrounding source-note text, AG order, date, `Source:` label, or trailing
`unless otherwise noted`. For `authority.ag_order_number`, retain the complete
visible `AG Order No. {number}` label rather than returning only its number.
Treat the earliest substantive `35.200` content version as the operative
amendment/effective date for this trace. Do not substitute the API metadata's
later `latest_amendment_date`. Reproduce the seven complete post-change section
headings in regulatory order, including section symbols/numbers, final
punctuation, and the reserved range. Reproduce the five
exceptions in paragraph order using their italicized headings without the final
period. Use the canonical covered-content labels `web content` and `mobile apps`,
and the conformance labels `Level A` and `Level AA` in that order.
For `codification.before_subpart_heading` and
`codification.after_subpart_heading`, return only the complete heading beginning
with `Subpart H—`; exclude adjacent cross-reference and source-note text.
Use `NOT_STATED` when a requested metadata item or requirement is not stated in
the supplied sources. Use `NOT_FOUND` when the requested historical codified
item is absent from the supplied point-in-time view. Do not infer a RIN,
Federal Register document number, testing frequency, or pre-effective section
list from outside knowledge.
For every field named in the evidence requirements, cite the exact supplied URL
and a short exact quote from that source. The evidence for section headings,
covered content, and exceptions must include one distinct citation per returned
item in the same order. Do not search the broader web, follow external links,
sign in, subscribe, submit a comment, or take any state-changing action.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
REQUIRED OUTPUT JSON SCHEMA:
{
"$defs": {
"citation": {
"additionalProperties": false,
"properties": {
"quote": {
"minLength": 1,
"type": "string"
},
"url": {
"format": "uri",
"type": "string"
}
},
"required": [
"url",
"quote"
],
"type": "object"
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema",
"items": {
"additionalProperties": false,
"properties": {
"authority": {
"additionalProperties": false,
"properties": {
"ag_order_number": {
"description": "Complete visible AG Order No. {number} label, not only the numeric identifier.",
"type": "string"
},
"federal_register_document_number": {
"type": "string"
},
"publication_date": {
"format": "date",
"type": "string"
},
"rin": {
"type": "string"
},
"subpart_source_citation": {
"description": "Federal Register volume-and-page citation only, formatted as {volume} FR {page}; omit the surrounding source-note text, AG order, date, and labels.",
"type": "string"
}
},
"required": [
"subpart_source_citation",
"publication_date",
"ag_order_number",
"rin",
"federal_register_document_number"
],
"type": "object"
},
"codification": {
"additionalProperties": false,
"properties": {
"after_section_headings": {
"items": {
"type": "string"
},
"maxItems": 7,
"minItems": 7,
"type": "array"
},
"after_subpart_heading": {
"type": "string"
},
"before_sections": {
"type": "string"
},
"before_subpart_heading": {
"type": "string"
}
},
"required": [
"before_subpart_heading",
"before_sections",
"after_subpart_heading",
"after_section_headings"
],
"type": "object"
},
"evidence": {
"additionalProperties": false,
"properties": {
"authority.ag_order_number": {
"$ref": "#/$defs/citation"
},
"authority.publication_date": {
"$ref": "#/$defs/citation"
},
"authority.subpart_source_citation": {
"$ref": "#/$defs/citation"
},
"codification.after_section_headings": {
"items": {
"$ref": "#/$defs/citation"
},
"maxItems": 7,
"minItems": 7,
"type": "array"
},
"codification.after_subpart_heading": {
"$ref": "#/$defs/citation"
},
"codification.before_subpart_heading": {
"$ref": "#/$defs/citation"
},
"exceptions": {
"items": {
"$ref": "#/$defs/citation"
},
"maxItems": 5,
"minItems": 5,
"type": "array"
},
"requirements.covered_content": {
"items": {
"$ref": "#/$defs/citation"
},
"maxItems": 2,
"minItems": 2,
"type": "array"
},
"requirements.large_entity_compliance_date": {
"$ref": "#/$defs/citation"
},
"requirements.small_or_special_district_compliance_date": {
"$ref": "#/$defs/citation"
},
"requirements.standard": {
"$ref": "#/$defs/citation"
},
"version_trace.effective_date": {
"$ref": "#/$defs/citation"
}
},
"required": [
"authority.subpart_source_citation",
"authority.publication_date",
"authority.ag_order_number",
"version_trace.effective_date",
"codification.before_subpart_heading",
"codification.after_subpart_heading",
"codification.after_section_headings",
"requirements.covered_content",
"requirements.standard",
"requirements.large_entity_compliance_date",
"requirements.small_or_special_district_compliance_date",
"exceptions"
],
"type": "object"
},
"exceptions": {
"items": {
"type": "string"
},
"maxItems": 5,
"minItems": 5,
"type": "array"
},
"official_urls": {
"additionalProperties": false,
"properties": {
"after": {
"format": "uri",
"type": "string"
},
"before": {
"format": "uri",
"type": "string"
},
"versions": {
"format": "uri",
"type": "string"
}
},
"required": [
"before",
"after",
"versions"
],
"type": "object"
},
"requirements": {
"additionalProperties": false,
"properties": {
"conformance_levels": {
"items": {
"type": "string"
},
"maxItems": 2,
"minItems": 2,
"type": "array"
},
"covered_content": {
"items": {
"type": "string"
},
"maxItems": 2,
"minItems": 2,
"type": "array"
},
"large_entity_compliance_date": {
"format": "date",
"type": "string"
},
"mandatory_testing_frequency": {
"type": "string"
},
"small_or_special_district_compliance_date": {
"format": "date",
"type": "string"
},
"standard": {
"type": "string"
}
},
"required": [
"covered_content",
"standard",
"conformance_levels",
"large_entity_compliance_date",
"small_or_special_district_compliance_date",
"mandatory_testing_frequency"
],
"type": "object"
},
"version_trace": {
"additionalProperties": false,
"properties": {
"cfr_part": {
"type": "string"
},
"cfr_title": {
"type": "integer"
},
"comparison_after": {
"format": "date",
"type": "string"
},
"comparison_before": {
"format": "date",
"type": "string"
},
"effective_date": {
"format": "date",
"type": "string"
},
"first_substantive_version": {
"format": "date",
"type": "string"
},
"section": {
"type": "string"
}
},
"required": [
"cfr_title",
"cfr_part",
"section",
"comparison_before",
"comparison_after",
"effective_date",
"first_substantive_version"
],
"type": "object"
}
},
"required": [
"authority",
"version_trace",
"codification",
"requirements",
"exceptions",
"official_urls",
"evidence"
],
"type": "object"
},
"maxItems": 1,
"minItems": 1,
"type": "array"
}
FROZEN TASK INPUT:
# Frozen task input: 28 CFR part 35, subpart H
Trace the Federal Register source note and effective codification of the
Department of Justice's web and mobile accessibility provisions using only
these official eCFR endpoints:
- Before view (June 23, 2024): https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-23/title-28?chapter=I&part=35&subpart=H
- After view (June 24, 2024): https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-24/title-28?chapter=I&part=35&subpart=H
- Section 35.200 version history: https://www.ecfr.gov/api/versioner/v1/versions/title-28.json?part=35§ion=35.200
The before and after renderer endpoints are the fixed point-in-time comparison.
The version-history response may contain a later amendment; use the earliest
substantive version for section 35.200, not the metadata's latest date.
Return exactly these fields:
- `authority.subpart_source_citation`
- `authority.publication_date`
- `authority.ag_order_number`
- `authority.rin` (explicit silence trap)
- `authority.federal_register_document_number` (explicit silence trap)
- `version_trace.cfr_title`
- `version_trace.cfr_part`
- `version_trace.section`
- `version_trace.comparison_before`
- `version_trace.comparison_after`
- `version_trace.effective_date`
- `version_trace.first_substantive_version`
- `codification.before_subpart_heading`
- `codification.before_sections` (explicit absence trap)
- `codification.after_subpart_heading`
- `codification.after_section_headings`
- `requirements.covered_content`
- `requirements.standard`
- `requirements.conformance_levels`
- `requirements.large_entity_compliance_date`
- `requirements.small_or_special_district_compliance_date`
- `requirements.mandatory_testing_frequency` (explicit silence trap)
- `exceptions`
- `official_urls.before`
- `official_urls.after`
- `official_urls.versions`
- `evidence`
Evidence is a top-level object keyed by the exact dotted field path. Provide
evidence for:
- `authority.subpart_source_citation`
- `authority.publication_date`
- `authority.ag_order_number`
- `version_trace.effective_date`
- `codification.before_subpart_heading`
- `codification.after_subpart_heading`
- `codification.after_section_headings`
- `requirements.covered_content`
- `requirements.standard`
- `requirements.large_entity_compliance_date`
- `requirements.small_or_special_district_compliance_date`
- `exceptions`
Each ordinary evidence value is one `{"url": "...", "quote": "..."}` object.
Evidence for `codification.after_section_headings`,
`requirements.covered_content`, and `exceptions` is instead an array containing
exactly 7, 2, and 5 citation objects respectively, one per returned item in the
same order. Do not cite any URL other than the three exact URLs above.
Expected output records — evaluator only
[
{
"authority": {
"subpart_source_citation": "89 FR 31337",
"publication_date": "2024-04-24",
"ag_order_number": "AG Order No. 5919-2024",
"rin": "NOT_STATED",
"federal_register_document_number": "NOT_STATED"
},
"version_trace": {
"cfr_title": 28,
"cfr_part": "35",
"section": "35.200",
"comparison_before": "2024-06-23",
"comparison_after": "2024-06-24",
"effective_date": "2024-06-24",
"first_substantive_version": "2024-06-24"
},
"codification": {
"before_subpart_heading": "Subpart H—General",
"before_sections": "NOT_FOUND",
"after_subpart_heading": "Subpart H—Web and Mobile Accessibility",
"after_section_headings": [
"§ 35.200 Requirements for web and mobile accessibility.",
"§ 35.201 Exceptions.",
"§ 35.202 Conforming alternate versions.",
"§ 35.203 Equivalent facilitation.",
"§ 35.204 Duties.",
"§ 35.205 Effect of noncompliance that has a minimal impact on access.",
"§§ 35.206-35.209 [Reserved]"
]
},
"requirements": {
"covered_content": [
"web content",
"mobile apps"
],
"standard": "WCAG 2.1",
"conformance_levels": [
"Level A",
"Level AA"
],
"large_entity_compliance_date": "2026-04-24",
"small_or_special_district_compliance_date": "2027-04-26",
"mandatory_testing_frequency": "NOT_STATED"
},
"exceptions": [
"Archived web content",
"Preexisting conventional electronic documents",
"Content posted by a third party",
"Individualized, password-protected or otherwise secured conventional electronic documents",
"Preexisting social media posts"
],
"official_urls": {
"before": "https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-23/title-28?chapter=I&part=35&subpart=H",
"after": "https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-24/title-28?chapter=I&part=35&subpart=H",
"versions": "https://www.ecfr.gov/api/versioner/v1/versions/title-28.json?part=35§ion=35.200"
}
}
]Field comparison and evidence rules
{
"record_key": "version_trace.section",
"success_threshold": "0.90",
"evidence_threshold": "1.00",
"field_rules": [
{
"name": "authority.subpart_source_citation",
"kind": "exact",
"critical": true,
"evidence_source": "after"
},
{
"name": "authority.publication_date",
"kind": "date",
"evidence_source": "after"
},
{
"name": "authority.ag_order_number",
"kind": "exact",
"evidence_source": "after"
},
{
"name": "authority.rin",
"kind": "exact",
"critical": true
},
{
"name": "authority.federal_register_document_number",
"kind": "exact",
"critical": true
},
{
"name": "version_trace.cfr_title",
"kind": "integer"
},
{
"name": "version_trace.cfr_part",
"kind": "exact"
},
{
"name": "version_trace.section",
"kind": "exact"
},
{
"name": "version_trace.comparison_before",
"kind": "date"
},
{
"name": "version_trace.comparison_after",
"kind": "date"
},
{
"name": "version_trace.effective_date",
"kind": "date",
"critical": true,
"evidence_source": "versions"
},
{
"name": "version_trace.first_substantive_version",
"kind": "date"
},
{
"name": "codification.before_subpart_heading",
"kind": "exact",
"evidence_source": "before"
},
{
"name": "codification.before_sections",
"kind": "exact",
"critical": true
},
{
"name": "codification.after_subpart_heading",
"kind": "exact",
"critical": true,
"evidence_source": "after"
},
{
"name": "codification.after_section_headings",
"kind": "ordered_list",
"critical": true,
"evidence_source": "after_sections"
},
{
"name": "requirements.covered_content",
"kind": "ordered_list",
"critical": true,
"evidence_source": "after_covered_content"
},
{
"name": "requirements.standard",
"kind": "exact",
"critical": true,
"evidence_source": "after"
},
{
"name": "requirements.conformance_levels",
"kind": "ordered_list",
"critical": true
},
{
"name": "requirements.large_entity_compliance_date",
"kind": "date",
"critical": true,
"evidence_source": "after"
},
{
"name": "requirements.small_or_special_district_compliance_date",
"kind": "date",
"critical": true,
"evidence_source": "after"
},
{
"name": "requirements.mandatory_testing_frequency",
"kind": "exact",
"critical": true
},
{
"name": "exceptions",
"kind": "ordered_list",
"critical": true,
"evidence_source": "after_exceptions"
},
{
"name": "official_urls.before",
"kind": "exact"
},
{
"name": "official_urls.after",
"kind": "exact"
},
{
"name": "official_urls.versions",
"kind": "exact"
}
],
"evidence_sources": {
"before": {
"url": "https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-23/title-28?chapter=I&part=35&subpart=H",
"path": "fixtures/ecfr-2024-06-23.html",
"format": "html",
"require_quote": true,
"normalize_quote_whitespace": true
},
"after": {
"url": "https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-24/title-28?chapter=I&part=35&subpart=H",
"path": "fixtures/ecfr-2024-06-24.html",
"format": "html",
"require_quote": true,
"normalize_quote_whitespace": true
},
"after_sections": {
"url": "https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-24/title-28?chapter=I&part=35&subpart=H",
"path": "fixtures/ecfr-2024-06-24.html",
"format": "html",
"min_citations": 7,
"require_quote": true,
"normalize_quote_whitespace": true
},
"after_covered_content": {
"url": "https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-24/title-28?chapter=I&part=35&subpart=H",
"path": "fixtures/ecfr-2024-06-24.html",
"format": "html",
"min_citations": 2,
"require_quote": true,
"normalize_quote_whitespace": true
},
"after_exceptions": {
"url": "https://www.ecfr.gov/api/renderer/v1/content/enhanced/2024-06-24/title-28?chapter=I&part=35&subpart=H",
"path": "fixtures/ecfr-2024-06-24.html",
"format": "html",
"min_citations": 5,
"require_quote": true,
"normalize_quote_whitespace": true
},
"versions": {
"url": "https://www.ecfr.gov/api/versioner/v1/versions/title-28.json?part=35§ion=35.200",
"path": "fixtures/ecfr-35.200-versions.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
}
}
}Addenda trace
Construction addenda and deadline-change monitoring. Pack 6; historical execution pack.
Executed by: Claude Fable 5.1, Claude Opus 5, Claude Sonnet 5, GPT-5.6 Sol, Gemini 3.8 Flash, Grok 4.6.
Time limit: 15 minutes.
Supplied source URLs
- https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view
- https://sam.gov/api/prod/opps/v2/opportunities/dfc5bd2c65ae4a4bb910a395175a3343?latest=true
- https://sam.gov/api/prod/opps/v2/opportunities/70d3a1854d2f445a83e9bcbdcda10221
- https://sam.gov/api/prod/opps/v2/opportunities/50bcb3f873654600a10619194aba8640
- https://sam.gov/api/prod/opps/v3/opportunities/dfc5bd2c65ae4a4bb910a395175a3343/resources
Task prompt template
Reconcile the supplied construction solicitation history using only the supplied official public SAM.gov URLs and the Kernel browser. Return exactly one record matching the supplied output contract. Treat the version posted June 25 as the baseline immediately before Amendment 0001, the June 26 version as the state after Amendment 0001, and the latest July 2 version as the current state after Amendment 0002. Preserve all three deadlines and calculate the total calendar-date extension from June 29 to July 10. Do not silently replace earlier dates. Report `identity.lifecycle_status` at the synthetic monitoring timestamp, `2026-07-08T00:00:00Z`. At that time the July 10 response deadline was still open, the notice had not been cancelled, and its July 25 automatic archive date had not arrived, so the lifecycle status is `ACTIVE`. Do not back-propagate the later archived state recorded when the frozen source was fetched. Report only scope changes stated in the supplied notice history. The notice states that Amendment 0001 published an updated SOW but does not enumerate its specific changes; use `NOT_STATED` for those specifics. Amendment 0002 names project areas 2, 5, 6, and 7 but does not state the numeric window-count deltas; preserve the areas and use `NOT_STATED` for the missing deltas. Copy attachment filenames exactly and order amendment and SOW filenames from oldest to newest. The canonical output preserves the source spelling and punctuation; scoring ignores only differences in runs of whitespace inside filename fields. For the acknowledgment list, use bare amendment identifiers `0001` and `0002` in chronological order. The equivalent prefixed forms `Amendment 0001` and `Amendment 0002` are accepted, but order is still significant. The place-of-performance state may be `WI` or `Wisconsin`. Use the missing-value tokens exactly as follows; do not interchange them: - `commercial.award_amount` is `NOT_FOUND` because that concrete record value is absent from the complete official data structures. - `scope_changes.amendment_0001_specific_scope_changes`, `scope_changes.amendment_0002_exact_window_count_deltas`, `site_visit.revised_site_visit_date`, `questions.revised_due_at`, and `commercial.bonding_requirement` are `NOT_STATED` because the complete supplied source set does not state those requested narrative facts. Do not correct the stated `2025` site-visit registration year; preserve it and flag the conflict with the 2026 solicitation chronology. For the synthetic monitoring decision, use `2026-07-08T00:00:00Z`. Mark `URGENT_REVIEW` when the current response window is open, its due calendar date is within three calendar dates of the as-of date, and a deadline or scope change is present. The permitted change categories are `DEADLINE`, `SCOPE`, `DOCUMENTS`, and `ACKNOWLEDGMENT`. Every field named in the evidence requirements must cite one complete URL from the supplied list whose frozen content directly supports that field, together with a short literal quote found in that same source. Several fields are supported by more than one supplied source; any reviewed supporting source is valid. Preserve each URL exactly, including its query string: do not decode, rewrite, shorten, append to, or otherwise modify it. Quote matching may normalize whitespace only. In particular, the notice page and the three reviewed notice-data versions are valid alternatives for `commercial.funds_currently_available` when the cited source contains the literal supporting sentence. Do not download attachments, acknowledge an amendment, submit an offer, register as a bidder, send an email, sign in, click, type, press a key, or take any state-changing action. Navigate directly to each complete supplied URL. Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and `BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
Reconstructed runner prompt and output contract
BENCHMARK EXECUTION CONTROL (mandatory):
- Use browser tools for the browser work in this task.
- Every browser_start_session call must explicitly pass provider="kernel".
- Start with the supplied reserved ephemeral site_key and pass
clear_cookies=false exactly. The run plan generated this never-before-issued
profile key so the browser starts empty without triggering asynchronous
deletion of another measured attempt's profile.
- Call browser_start_session exactly once. If that start fails, report the
benchmark error and stop instead of retrying with the same or another key.
- Never omit provider and never request Browserbase or automatic fallback.
- Inspect each browser session result. If it identifies any provider other than
Kernel, stop immediately and return a benchmark error instead of continuing.
- When the browser work is complete, call browser_close with neither session_id
nor close_all so the current chat's session is released without an approval.
- Do not take any state-changing action except the task card's explicitly
allowed benchmark-owned Save draft action.
TASK-SPECIFIC BROWSER IDENTITY (mandatory):
- Every browser_start_session call must pass site_key="benchmark-profile-placeholder" exactly. It is an ephemeral profile identifier frozen in this run plan; do not turn it into a URL or Markdown link.
- Every browser_start_session call must pass clear_cookies=false exactly.
- The only permitted tool names for this task are: list_tools, browser_start_session, browser_navigate, browser_snapshot, browser_scroll, browser_open_tab, browser_switch_tab, browser_close_tab, browser_close.
- The measured tool-invocation counts must satisfy: browser_close: exactly 1; browser_start_session: exactly 1.
- Every browser_start_session call must pass url="https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view" exactly. This is the canonical start URL for the task.
- URL-bearing browser calls may use only these exact resource URLs: https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view; https://sam.gov/api/prod/opps/v2/opportunities/dfc5bd2c65ae4a4bb910a395175a3343?latest=true; https://sam.gov/api/prod/opps/v2/opportunities/70d3a1854d2f445a83e9bcbdcda10221; https://sam.gov/api/prod/opps/v2/opportunities/50bcb3f873654600a10619194aba8640; https://sam.gov/api/prod/opps/v3/opportunities/dfc5bd2c65ae4a4bb910a395175a3343/resources. A fragment-only page or anchor selector is allowed on a listed resource; do not alter its scheme, host, path, or query.
- You must visit every listed URL using at least one URL-bearing browser call; opening only a subset fails the measured run.
Reconcile the supplied construction solicitation history using only the
supplied official public SAM.gov URLs and the Kernel browser. Return exactly
one record matching the supplied output contract.
Treat the version posted June 25 as the baseline immediately before Amendment
0001, the June 26 version as the state after Amendment 0001, and the latest
July 2 version as the current state after Amendment 0002. Preserve all three
deadlines and calculate the total calendar-date extension from June 29 to July
10. Do not silently replace earlier dates.
Report `identity.lifecycle_status` at the synthetic monitoring timestamp,
`2026-07-08T00:00:00Z`. At that time the July 10 response deadline was still
open, the notice had not been cancelled, and its July 25 automatic archive date
had not arrived, so the lifecycle status is `ACTIVE`. Do not back-propagate the
later archived state recorded when the frozen source was fetched.
Report only scope changes stated in the supplied notice history. The notice
states that Amendment 0001 published an updated SOW but does not enumerate its
specific changes; use `NOT_STATED` for those specifics. Amendment 0002 names
project areas 2, 5, 6, and 7 but does not state the numeric window-count deltas;
preserve the areas and use `NOT_STATED` for the missing deltas.
Copy attachment filenames exactly and order amendment and SOW filenames from
oldest to newest. The canonical output preserves the source spelling and
punctuation; scoring ignores only differences in runs of whitespace inside
filename fields. For the acknowledgment list, use bare amendment identifiers
`0001` and `0002` in chronological order. The equivalent prefixed forms
`Amendment 0001` and `Amendment 0002` are accepted, but order is still
significant. The place-of-performance state may be `WI` or `Wisconsin`.
Use the missing-value tokens exactly as follows; do not interchange them:
- `commercial.award_amount` is `NOT_FOUND` because that concrete record value
is absent from the complete official data structures.
- `scope_changes.amendment_0001_specific_scope_changes`,
`scope_changes.amendment_0002_exact_window_count_deltas`,
`site_visit.revised_site_visit_date`, `questions.revised_due_at`, and
`commercial.bonding_requirement` are `NOT_STATED` because the complete
supplied source set does not state those requested narrative facts.
Do not correct the stated `2025` site-visit registration year; preserve it and
flag the conflict with the 2026 solicitation chronology.
For the synthetic monitoring decision, use `2026-07-08T00:00:00Z`. Mark
`URGENT_REVIEW` when the current response window is open, its due calendar date
is within three calendar dates of the as-of date, and a deadline or scope change
is present. The permitted change categories are `DEADLINE`, `SCOPE`,
`DOCUMENTS`, and `ACKNOWLEDGMENT`.
Every field named in the evidence requirements must cite one complete URL from
the supplied list whose frozen content directly supports that field, together
with a short literal quote found in that same source. Several fields are
supported by more than one supplied source; any reviewed supporting source is
valid. Preserve each URL exactly, including its query string: do not decode,
rewrite, shorten, append to, or otherwise modify it. Quote matching may
normalize whitespace only. In particular, the notice page and the three
reviewed notice-data versions are valid alternatives for
`commercial.funds_currently_available` when the cited source contains the
literal supporting sentence. Do not download attachments, acknowledge an
amendment, submit an offer, register as a bidder, send an email, sign in, click,
type, press a key, or take any state-changing action. Navigate directly to each
complete supplied URL.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
REQUIRED OUTPUT JSON SCHEMA:
{
"$defs": {
"citation": {
"additionalProperties": false,
"properties": {
"quote": {
"minLength": 1,
"type": "string"
},
"url": {
"format": "uri",
"type": "string"
}
},
"required": [
"url",
"quote"
],
"type": "object"
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema",
"items": {
"additionalProperties": false,
"properties": {
"classification": {
"additionalProperties": false,
"properties": {
"naics_code": {
"type": "string"
},
"set_aside": {
"enum": [
"TOTAL_SMALL_BUSINESS",
"UNRESTRICTED",
"OTHER",
"NOT_STATED"
]
}
},
"required": [
"naics_code",
"set_aside"
],
"type": "object"
},
"commercial": {
"additionalProperties": false,
"properties": {
"award_amount": {
"description": "Use NOT_FOUND for this frozen case because the concrete award value is absent from the official data structures.",
"type": [
"string",
"number"
]
},
"bonding_requirement": {
"description": "Use NOT_STATED for this frozen case.",
"type": "string"
},
"funds_currently_available": {
"description": "Whether funds are presently available; the exact notice-page statement is valid evidence.",
"type": "boolean"
}
},
"required": [
"award_amount",
"bonding_requirement",
"funds_currently_available"
],
"type": "object"
},
"document_control": {
"additionalProperties": false,
"properties": {
"amendment_filenames": {
"description": "Canonical source filenames oldest-to-newest. Only runs of whitespace are comparison-normalized; spelling and punctuation are significant.",
"items": {
"type": "string"
},
"type": "array"
},
"attachment_count": {
"minimum": 0,
"type": "integer"
},
"latest_sow_filename": {
"description": "Canonical exact source filename. Only runs of whitespace are comparison-normalized; spelling and punctuation are significant.",
"type": "string"
},
"sow_filenames": {
"description": "Canonical source filenames oldest-to-newest. Only runs of whitespace are comparison-normalized; spelling and punctuation are significant.",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"attachment_count",
"amendment_filenames",
"sow_filenames",
"latest_sow_filename"
],
"type": "object"
},
"evidence": {
"additionalProperties": {
"$ref": "#/$defs/citation"
},
"type": "object"
},
"identity": {
"additionalProperties": false,
"properties": {
"current_version_id": {
"type": "string"
},
"lifecycle_status": {
"description": "Lifecycle status at monitoring.as_of_at (2026-07-08T00:00:00Z), not the later status when the frozen source was fetched.",
"enum": [
"ACTIVE",
"ARCHIVED",
"CANCELLED"
]
},
"solicitation_number": {
"type": "string"
},
"title": {
"type": "string"
}
},
"required": [
"title",
"solicitation_number",
"current_version_id",
"lifecycle_status"
],
"type": "object"
},
"monitoring": {
"additionalProperties": false,
"properties": {
"as_of_at": {
"format": "date-time",
"type": "string"
},
"change_categories": {
"items": {
"enum": [
"DEADLINE",
"SCOPE",
"DOCUMENTS",
"ACKNOWLEDGMENT"
]
},
"type": "array"
},
"current_due_date_open": {
"type": "boolean"
},
"deadline_extension_preserved_response_window": {
"type": "boolean"
},
"prior_due_date_passed": {
"type": "boolean"
},
"review_priority": {
"enum": [
"URGENT_REVIEW",
"NORMAL_REVIEW",
"NO_ACTION"
]
}
},
"required": [
"as_of_at",
"prior_due_date_passed",
"current_due_date_open",
"deadline_extension_preserved_response_window",
"change_categories",
"review_priority"
],
"type": "object"
},
"official_urls": {
"additionalProperties": false,
"properties": {
"after_amendment_0001_version": {
"format": "uri",
"type": "string"
},
"attachment_index": {
"format": "uri",
"type": "string"
},
"baseline_version": {
"format": "uri",
"type": "string"
},
"current_notice_data": {
"format": "uri",
"type": "string"
},
"notice": {
"format": "uri",
"type": "string"
}
},
"required": [
"notice",
"current_notice_data",
"baseline_version",
"after_amendment_0001_version",
"attachment_index"
],
"type": "object"
},
"place": {
"additionalProperties": false,
"properties": {
"city": {
"type": "string"
},
"postal_code": {
"type": "string"
},
"state": {
"description": "Place-of-performance state. WI and Wisconsin are equivalent.",
"type": "string"
}
},
"required": [
"city",
"state",
"postal_code"
],
"type": "object"
},
"questions": {
"additionalProperties": false,
"properties": {
"original_due_at": {
"format": "date-time",
"type": "string"
},
"revised_due_at": {
"description": "Use NOT_STATED for this frozen case.",
"type": "string"
}
},
"required": [
"original_due_at",
"revised_due_at"
],
"type": "object"
},
"scope_changes": {
"additionalProperties": false,
"properties": {
"amendment_0001_specific_scope_changes": {
"description": "Use NOT_STATED for this frozen case; the supplied history does not enumerate the specific Amendment 0001 changes.",
"type": "string"
},
"amendment_0001_updated_sow_published": {
"type": "boolean"
},
"amendment_0002_changed_project_areas": {
"items": {
"type": "integer"
},
"type": "array"
},
"amendment_0002_exact_window_count_deltas": {
"description": "Use NOT_STATED for this frozen case; the supplied history names areas but gives no numeric count deltas.",
"type": "string"
},
"amendment_0002_updated_sow_published": {
"type": "boolean"
}
},
"required": [
"amendment_0001_updated_sow_published",
"amendment_0001_specific_scope_changes",
"amendment_0002_changed_project_areas",
"amendment_0002_exact_window_count_deltas",
"amendment_0002_updated_sow_published"
],
"type": "object"
},
"site_visit": {
"additionalProperties": false,
"properties": {
"location": {
"type": "string"
},
"registration_deadline_as_stated": {
"type": "string"
},
"registration_year_conflict": {
"type": "boolean"
},
"revised_site_visit_date": {
"description": "Use NOT_STATED for this frozen case.",
"type": "string"
},
"scheduled_date_as_stated": {
"type": "string"
},
"scheduled_time_as_stated": {
"type": "string"
}
},
"required": [
"scheduled_date_as_stated",
"scheduled_time_as_stated",
"location",
"registration_deadline_as_stated",
"registration_year_conflict",
"revised_site_visit_date"
],
"type": "object"
},
"timeline": {
"additionalProperties": false,
"properties": {
"after_amendment_0001_due_at": {
"format": "date-time",
"type": "string"
},
"amendment_0001_posted_at": {
"format": "date-time",
"type": "string"
},
"amendment_0001_version_id": {
"type": "string"
},
"amendments_requiring_acknowledgment": {
"description": "Oldest-to-newest amendment identifiers. Bare 0001/0002 and corresponding Amendment-prefixed forms are equivalent; order remains significant.",
"items": {
"type": "string"
},
"type": "array"
},
"baseline_due_at": {
"format": "date-time",
"type": "string"
},
"baseline_posted_at": {
"format": "date-time",
"type": "string"
},
"baseline_version_id": {
"type": "string"
},
"current_due_at": {
"format": "date-time",
"type": "string"
},
"current_posted_at": {
"format": "date-time",
"type": "string"
},
"deadline_changes_count": {
"minimum": 0,
"type": "integer"
},
"deadline_sequence": {
"items": {
"format": "date",
"type": "string"
},
"type": "array"
},
"total_extension_calendar_days": {
"type": "integer"
}
},
"required": [
"baseline_version_id",
"baseline_posted_at",
"baseline_due_at",
"amendment_0001_version_id",
"amendment_0001_posted_at",
"after_amendment_0001_due_at",
"current_posted_at",
"current_due_at",
"deadline_sequence",
"deadline_changes_count",
"total_extension_calendar_days",
"amendments_requiring_acknowledgment"
],
"type": "object"
}
},
"required": [
"identity",
"classification",
"place",
"timeline",
"scope_changes",
"site_visit",
"questions",
"document_control",
"commercial",
"monitoring",
"official_urls",
"evidence"
],
"type": "object"
},
"maxItems": 1,
"minItems": 1,
"type": "array"
}
FROZEN TASK INPUT:
# Frozen task input: construction addenda and deadline-change monitoring
Reconcile SAM.gov solicitation `W50S9H26QA018`, *Volk Field Billeting Blind
Replacement – Multiple Buildings*, as of `2026-07-08T00:00:00Z`.
Use only these official public sources:
- Notice page: https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view
- Current notice data: https://sam.gov/api/prod/opps/v2/opportunities/dfc5bd2c65ae4a4bb910a395175a3343?latest=true
- Baseline before Amendment 0001: https://sam.gov/api/prod/opps/v2/opportunities/70d3a1854d2f445a83e9bcbdcda10221
- State after Amendment 0001: https://sam.gov/api/prod/opps/v2/opportunities/50bcb3f873654600a10619194aba8640
- Current attachment index: https://sam.gov/api/prod/opps/v3/opportunities/dfc5bd2c65ae4a4bb910a395175a3343/resources
Return every field defined by `output.schema.json`. Evidence is a top-level
object keyed by exact dotted field path. Provide one exact-URL quote for:
- `identity.solicitation_number`
- `timeline.baseline_due_at`
- `timeline.after_amendment_0001_due_at`
- `timeline.current_due_at`
- `timeline.amendments_requiring_acknowledgment`
- `scope_changes.amendment_0002_changed_project_areas`
- `site_visit.registration_deadline_as_stated`
- `questions.original_due_at`
- `document_control.latest_sow_filename`
- `commercial.funds_currently_available`
Each evidence value must be `{"url": "...", "quote": "..."}`. Cite any exact
URL above whose frozen content directly supports the complete field value and
contains the literal quote. Preserve the URL exactly, including its query
string; do not decode, rewrite, shorten, append to, or otherwise modify it.
Whitespace normalization is allowed when matching quotes. For
`commercial.funds_currently_available`, the notice page, current notice data,
baseline version, and state after Amendment 0001 are the reviewed alternatives;
no other source is accepted for that field.
Output conventions for this frozen case:
- `place.state` may be `WI` or `Wisconsin`.
- `identity.lifecycle_status` is the status at `2026-07-08T00:00:00Z`, not the
later status when the source snapshot was fetched. The response deadline was
still open, the notice was not cancelled, and automatic archival was dated
July 25, so use `ACTIVE`.
- Keep amendment acknowledgments oldest-to-newest. Bare IDs `0001`, `0002` are
canonical; the corresponding `Amendment 0001`, `Amendment 0002` forms are
equivalent, but reversed order is not.
- Preserve filename spelling, punctuation, and order. Only runs of whitespace
inside filename fields are treated as equivalent.
- Use `NOT_FOUND` only for `commercial.award_amount`.
- Use `NOT_STATED` for
`scope_changes.amendment_0001_specific_scope_changes`,
`scope_changes.amendment_0002_exact_window_count_deltas`,
`site_visit.revised_site_visit_date`, `questions.revised_due_at`, and
`commercial.bonding_requirement`. Do not swap the two missing-value tokens.
Expected output records — evaluator only
[
{
"identity": {
"title": "VOLK FIELD BILLETING BLIND REPLACEMENT - MULTIPLE BUILDINGS",
"solicitation_number": "W50S9H26QA018",
"current_version_id": "dfc5bd2c65ae4a4bb910a395175a3343",
"lifecycle_status": "ACTIVE"
},
"classification": {
"naics_code": "337920",
"set_aside": "TOTAL_SMALL_BUSINESS"
},
"place": {
"city": "Camp Douglas",
"state": "WI",
"postal_code": "54618"
},
"timeline": {
"baseline_version_id": "70d3a1854d2f445a83e9bcbdcda10221",
"baseline_posted_at": "2026-06-25T16:07:33.058Z",
"baseline_due_at": "2026-06-29T13:00:00-05:00",
"amendment_0001_version_id": "50bcb3f873654600a10619194aba8640",
"amendment_0001_posted_at": "2026-06-26T13:26:26.050Z",
"after_amendment_0001_due_at": "2026-07-06T13:00:00-05:00",
"current_posted_at": "2026-07-02T14:40:49.843Z",
"current_due_at": "2026-07-10T13:00:00-05:00",
"deadline_sequence": [
"2026-06-29",
"2026-07-06",
"2026-07-10"
],
"deadline_changes_count": 2,
"total_extension_calendar_days": 11,
"amendments_requiring_acknowledgment": [
"0001",
"0002"
]
},
"scope_changes": {
"amendment_0001_updated_sow_published": true,
"amendment_0001_specific_scope_changes": "NOT_STATED",
"amendment_0002_changed_project_areas": [
2,
5,
6,
7
],
"amendment_0002_exact_window_count_deltas": "NOT_STATED",
"amendment_0002_updated_sow_published": true
},
"site_visit": {
"scheduled_date_as_stated": "16 JUNE",
"scheduled_time_as_stated": "9:00 AM CDT",
"location": "100 Independence Dr. Camp Douglas, WI 54618, BLDG 475",
"registration_deadline_as_stated": "9 JUNE 2025 at 11:00 AM CDT",
"registration_year_conflict": true,
"revised_site_visit_date": "NOT_STATED"
},
"questions": {
"original_due_at": "2026-06-22T13:00:00-05:00",
"revised_due_at": "NOT_STATED"
},
"document_control": {
"attachment_count": 10,
"amendment_filenames": [
"SOLICITATION AMENDMENT 0001 - W50S9H26QA0180001 SF 30.pdf",
"SOLICITATION AMENDMENT 0002 - W50S9H26QA0180002 SF 30.pdf"
],
"sow_filenames": [
"STATEMENT OF WORK - VOLK FIELD BILLETING BLIND REPLACEMENT - MULTIPLE BUILDINGS.pdf",
"STATEMENT OF WORK - LODGING FACILITY BLIND REPLACEMENT - UPDATED 6.25.2026.pdf",
"STATEMENT OF WORK - LODGING FACILITY BLIND REPLACEMENT - UPDATED 7.2.2026.pdf"
],
"latest_sow_filename": "STATEMENT OF WORK - LODGING FACILITY BLIND REPLACEMENT - UPDATED 7.2.2026.pdf"
},
"commercial": {
"award_amount": "NOT_FOUND",
"bonding_requirement": "NOT_STATED",
"funds_currently_available": false
},
"monitoring": {
"as_of_at": "2026-07-08T00:00:00Z",
"prior_due_date_passed": true,
"current_due_date_open": true,
"deadline_extension_preserved_response_window": true,
"change_categories": [
"DEADLINE",
"SCOPE",
"DOCUMENTS",
"ACKNOWLEDGMENT"
],
"review_priority": "URGENT_REVIEW"
},
"official_urls": {
"notice": "https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view",
"current_notice_data": "https://sam.gov/api/prod/opps/v2/opportunities/dfc5bd2c65ae4a4bb910a395175a3343?latest=true",
"baseline_version": "https://sam.gov/api/prod/opps/v2/opportunities/70d3a1854d2f445a83e9bcbdcda10221",
"after_amendment_0001_version": "https://sam.gov/api/prod/opps/v2/opportunities/50bcb3f873654600a10619194aba8640",
"attachment_index": "https://sam.gov/api/prod/opps/v3/opportunities/dfc5bd2c65ae4a4bb910a395175a3343/resources"
}
}
]Field comparison and evidence rules
{
"record_key": "identity.solicitation_number",
"success_threshold": "0.90",
"evidence_threshold": "1.00",
"field_rules": [
{
"name": "identity.title",
"kind": "exact"
},
{
"name": "identity.solicitation_number",
"kind": "exact",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data",
"baseline",
"amendment_0001"
]
},
{
"name": "identity.current_version_id",
"kind": "exact"
},
{
"name": "identity.lifecycle_status",
"kind": "enum",
"enum_values": [
"ACTIVE",
"ARCHIVED",
"CANCELLED"
]
},
{
"name": "classification.naics_code",
"kind": "exact"
},
{
"name": "classification.set_aside",
"kind": "enum",
"enum_values": [
"TOTAL_SMALL_BUSINESS",
"UNRESTRICTED",
"OTHER",
"NOT_STATED"
]
},
{
"name": "place.city",
"kind": "exact"
},
{
"name": "place.state",
"kind": "aliases",
"aliases": {
"Wisconsin": "WI"
}
},
{
"name": "place.postal_code",
"kind": "exact"
},
{
"name": "timeline.baseline_version_id",
"kind": "exact"
},
{
"name": "timeline.baseline_posted_at",
"kind": "datetime"
},
{
"name": "timeline.baseline_due_at",
"kind": "datetime",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data",
"baseline",
"amendment_0001"
]
},
{
"name": "timeline.amendment_0001_version_id",
"kind": "exact"
},
{
"name": "timeline.amendment_0001_posted_at",
"kind": "datetime"
},
{
"name": "timeline.after_amendment_0001_due_at",
"kind": "datetime",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data",
"amendment_0001"
]
},
{
"name": "timeline.current_posted_at",
"kind": "datetime"
},
{
"name": "timeline.current_due_at",
"kind": "datetime",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data"
]
},
{
"name": "timeline.deadline_sequence",
"kind": "ordered_list",
"element_kind": "date",
"critical": true
},
{
"name": "timeline.deadline_changes_count",
"kind": "integer"
},
{
"name": "timeline.total_extension_calendar_days",
"kind": "integer",
"critical": true
},
{
"name": "timeline.amendments_requiring_acknowledgment",
"kind": "ordered_list",
"element_kind": "aliases",
"element_aliases": {
"AMENDMENT 0001": "0001",
"Amendment 0001": "0001",
"amendment 0001": "0001",
"AMENDMENT 0002": "0002",
"Amendment 0002": "0002",
"amendment 0002": "0002"
},
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data"
]
},
{
"name": "scope_changes.amendment_0001_updated_sow_published",
"kind": "exact"
},
{
"name": "scope_changes.amendment_0001_specific_scope_changes",
"kind": "exact",
"critical": true
},
{
"name": "scope_changes.amendment_0002_changed_project_areas",
"kind": "ordered_list",
"element_kind": "integer",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data"
]
},
{
"name": "scope_changes.amendment_0002_exact_window_count_deltas",
"kind": "exact",
"critical": true
},
{
"name": "scope_changes.amendment_0002_updated_sow_published",
"kind": "exact"
},
{
"name": "site_visit.scheduled_date_as_stated",
"kind": "exact"
},
{
"name": "site_visit.scheduled_time_as_stated",
"kind": "aliases",
"aliases": {
"9:00 AM CDT": "09:00 CDT",
"09:00 AM CDT": "09:00 CDT"
}
},
{
"name": "site_visit.location",
"kind": "exact"
},
{
"name": "site_visit.registration_deadline_as_stated",
"kind": "exact",
"evidence_sources": [
"notice",
"current_notice_data",
"baseline",
"amendment_0001"
]
},
{
"name": "site_visit.registration_year_conflict",
"kind": "exact",
"critical": true
},
{
"name": "site_visit.revised_site_visit_date",
"kind": "exact",
"critical": true
},
{
"name": "questions.original_due_at",
"kind": "datetime",
"evidence_sources": [
"notice",
"current_notice_data",
"baseline",
"amendment_0001"
]
},
{
"name": "questions.revised_due_at",
"kind": "exact",
"critical": true
},
{
"name": "document_control.attachment_count",
"kind": "integer"
},
{
"name": "document_control.amendment_filenames",
"kind": "ordered_list",
"element_kind": "whitespace_normalized"
},
{
"name": "document_control.sow_filenames",
"kind": "ordered_list",
"element_kind": "whitespace_normalized"
},
{
"name": "document_control.latest_sow_filename",
"kind": "whitespace_normalized",
"critical": true,
"evidence_source": "attachment_index"
},
{
"name": "commercial.award_amount",
"kind": "exact",
"critical": true
},
{
"name": "commercial.bonding_requirement",
"kind": "exact",
"critical": true
},
{
"name": "commercial.funds_currently_available",
"kind": "exact",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data",
"baseline",
"amendment_0001"
]
},
{
"name": "monitoring.as_of_at",
"kind": "datetime"
},
{
"name": "monitoring.prior_due_date_passed",
"kind": "exact"
},
{
"name": "monitoring.current_due_date_open",
"kind": "exact",
"critical": true
},
{
"name": "monitoring.deadline_extension_preserved_response_window",
"kind": "exact",
"critical": true
},
{
"name": "monitoring.change_categories",
"kind": "unordered_set",
"element_kind": "enum",
"element_enum_values": [
"DEADLINE",
"SCOPE",
"DOCUMENTS",
"ACKNOWLEDGMENT"
],
"critical": true
},
{
"name": "monitoring.review_priority",
"kind": "enum",
"enum_values": [
"URGENT_REVIEW",
"NORMAL_REVIEW",
"NO_ACTION"
],
"critical": true
},
{
"name": "official_urls.notice",
"kind": "exact"
},
{
"name": "official_urls.current_notice_data",
"kind": "exact"
},
{
"name": "official_urls.baseline_version",
"kind": "exact"
},
{
"name": "official_urls.after_amendment_0001_version",
"kind": "exact"
},
{
"name": "official_urls.attachment_index",
"kind": "exact"
}
],
"evidence_sources": {
"notice": {
"url": "https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view",
"path": "fixtures/sam-notice-visible.txt",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
},
"current_notice_data": {
"url": "https://sam.gov/api/prod/opps/v2/opportunities/dfc5bd2c65ae4a4bb910a395175a3343?latest=true",
"path": "fixtures/sam-notice.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
},
"baseline": {
"url": "https://sam.gov/api/prod/opps/v2/opportunities/70d3a1854d2f445a83e9bcbdcda10221",
"path": "fixtures/sam-version-original.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
},
"amendment_0001": {
"url": "https://sam.gov/api/prod/opps/v2/opportunities/50bcb3f873654600a10619194aba8640",
"path": "fixtures/sam-version-amendment-0001.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
},
"attachment_index": {
"url": "https://sam.gov/api/prod/opps/v3/opportunities/dfc5bd2c65ae4a4bb910a395175a3343/resources",
"path": "fixtures/sam-resources.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
}
}
}Addenda trace
Construction addenda and deadline-change monitoring. Pack 7; reviewed scoring pack.
Executed by: GPT-6 Astra, Kimi K3.
Time limit: 15 minutes.
Supplied source URLs
- https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view
- https://sam.gov/api/prod/opps/v2/opportunities/dfc5bd2c65ae4a4bb910a395175a3343?latest=true
- https://sam.gov/api/prod/opps/v2/opportunities/70d3a1854d2f445a83e9bcbdcda10221
- https://sam.gov/api/prod/opps/v2/opportunities/50bcb3f873654600a10619194aba8640
- https://sam.gov/api/prod/opps/v3/opportunities/dfc5bd2c65ae4a4bb910a395175a3343/resources
Task prompt template
Reconcile the supplied construction solicitation history using only the supplied official public SAM.gov URLs and the Kernel browser. Return exactly one record matching the supplied output contract. Treat the version posted June 25 as the baseline immediately before Amendment 0001, the June 26 version as the state after Amendment 0001, and the latest July 2 version as the current state after Amendment 0002. Preserve all three deadlines and calculate the total calendar-date extension from June 29 to July 10. Do not silently replace earlier dates. Report `identity.lifecycle_status` at the synthetic monitoring timestamp, `2026-07-08T00:00:00Z`. At that time the July 10 response deadline was still open, the notice had not been cancelled, and its July 25 automatic archive date had not arrived, so the lifecycle status is `ACTIVE`. Do not back-propagate the later archived state recorded when the frozen source was fetched. Report only scope changes stated in the supplied notice history. The notice states that Amendment 0001 published an updated SOW but does not enumerate its specific changes; use `NOT_STATED` for those specifics. Amendment 0002 names project areas 2, 5, 6, and 7 but does not state the numeric window-count deltas; preserve the areas and use `NOT_STATED` for the missing deltas. Copy attachment filenames exactly and order amendment and SOW filenames from oldest to newest. The canonical output preserves the source spelling and punctuation; scoring ignores only differences in runs of whitespace inside filename fields. For the acknowledgment list, use bare amendment identifiers `0001` and `0002` in chronological order. The equivalent prefixed forms `Amendment 0001` and `Amendment 0002` are accepted, but order is still significant. The place-of-performance state may be `WI` or `Wisconsin`. Use the missing-value tokens exactly as follows; do not interchange them: - `commercial.award_amount` is `NOT_FOUND` because that concrete record value is absent from the complete official data structures. - `scope_changes.amendment_0001_specific_scope_changes`, `scope_changes.amendment_0002_exact_window_count_deltas`, `site_visit.revised_site_visit_date`, `questions.revised_due_at`, and `commercial.bonding_requirement` are `NOT_STATED` because the complete supplied source set does not state those requested narrative facts. Do not correct the stated `2025` site-visit registration year; preserve it and flag the conflict with the 2026 solicitation chronology. For the synthetic monitoring decision, use `2026-07-08T00:00:00Z`. Mark `URGENT_REVIEW` when the current response window is open, its due calendar date is within three calendar dates of the as-of date, and a deadline or scope change is present. The permitted change categories are `DEADLINE`, `SCOPE`, `DOCUMENTS`, and `ACKNOWLEDGMENT`. Every field named in the evidence requirements must cite one complete URL from the supplied list whose frozen content directly supports that field, together with a short literal quote found in that same source. Several fields are supported by more than one supplied source; any reviewed supporting source is valid. Preserve each URL exactly, including its query string: do not decode, rewrite, shorten, append to, or otherwise modify it. Quote matching may normalize whitespace only. In particular, the notice page and the three reviewed notice-data versions are valid alternatives for `commercial.funds_currently_available` when the cited source contains the literal supporting sentence. Do not download attachments, acknowledge an amendment, submit an offer, register as a bidder, send an email, sign in, click, type, press a key, or take any state-changing action. Navigate directly to each complete supplied URL. Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and `BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
Reconstructed runner prompt and output contract
BENCHMARK EXECUTION CONTROL (mandatory):
- Use browser tools for the browser work in this task.
- Every browser_start_session call must explicitly pass provider="kernel".
- Start with the supplied reserved ephemeral site_key and pass
clear_cookies=false exactly. The run plan generated this never-before-issued
profile key so the browser starts empty without triggering asynchronous
deletion of another measured attempt's profile.
- Call browser_start_session exactly once. If that start fails, report the
benchmark error and stop instead of retrying with the same or another key.
- Never omit provider and never request Browserbase or automatic fallback.
- Inspect each browser session result. If it identifies any provider other than
Kernel, stop immediately and return a benchmark error instead of continuing.
- When the browser work is complete, call browser_close with neither session_id
nor close_all so the current chat's session is released without an approval.
- Do not take any state-changing action except the task card's explicitly
allowed benchmark-owned Save draft action.
TASK-SPECIFIC BROWSER IDENTITY (mandatory):
- Every browser_start_session call must pass site_key="benchmark-profile-placeholder" exactly. It is an ephemeral profile identifier frozen in this run plan; do not turn it into a URL or Markdown link.
- Every browser_start_session call must pass clear_cookies=false exactly.
- The only permitted tool names for this task are: list_tools, browser_start_session, browser_navigate, browser_snapshot, browser_scroll, browser_open_tab, browser_switch_tab, browser_close_tab, browser_close.
- The measured tool-invocation counts must satisfy: browser_close: exactly 1; browser_start_session: exactly 1.
- Every browser_start_session call must pass url="https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view" exactly. This is the canonical start URL for the task.
- URL-bearing browser calls may use only these exact resource URLs: https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view; https://sam.gov/api/prod/opps/v2/opportunities/dfc5bd2c65ae4a4bb910a395175a3343?latest=true; https://sam.gov/api/prod/opps/v2/opportunities/70d3a1854d2f445a83e9bcbdcda10221; https://sam.gov/api/prod/opps/v2/opportunities/50bcb3f873654600a10619194aba8640; https://sam.gov/api/prod/opps/v3/opportunities/dfc5bd2c65ae4a4bb910a395175a3343/resources. A fragment-only page or anchor selector is allowed on a listed resource; do not alter its scheme, host, path, or query.
- You must visit every listed URL using at least one URL-bearing browser call; opening only a subset fails the measured run.
Reconcile the supplied construction solicitation history using only the
supplied official public SAM.gov URLs and the Kernel browser. Return exactly
one record matching the supplied output contract.
Treat the version posted June 25 as the baseline immediately before Amendment
0001, the June 26 version as the state after Amendment 0001, and the latest
July 2 version as the current state after Amendment 0002. Preserve all three
deadlines and calculate the total calendar-date extension from June 29 to July
10. Do not silently replace earlier dates.
Report `identity.lifecycle_status` at the synthetic monitoring timestamp,
`2026-07-08T00:00:00Z`. At that time the July 10 response deadline was still
open, the notice had not been cancelled, and its July 25 automatic archive date
had not arrived, so the lifecycle status is `ACTIVE`. Do not back-propagate the
later archived state recorded when the frozen source was fetched.
Report only scope changes stated in the supplied notice history. The notice
states that Amendment 0001 published an updated SOW but does not enumerate its
specific changes; use `NOT_STATED` for those specifics. Amendment 0002 names
project areas 2, 5, 6, and 7 but does not state the numeric window-count deltas;
preserve the areas and use `NOT_STATED` for the missing deltas.
Copy attachment filenames exactly and order amendment and SOW filenames from
oldest to newest. The canonical output preserves the source spelling and
punctuation; scoring ignores only differences in runs of whitespace inside
filename fields. For the acknowledgment list, use bare amendment identifiers
`0001` and `0002` in chronological order. The equivalent prefixed forms
`Amendment 0001` and `Amendment 0002` are accepted, but order is still
significant. The place-of-performance state may be `WI` or `Wisconsin`.
Use the missing-value tokens exactly as follows; do not interchange them:
- `commercial.award_amount` is `NOT_FOUND` because that concrete record value
is absent from the complete official data structures.
- `scope_changes.amendment_0001_specific_scope_changes`,
`scope_changes.amendment_0002_exact_window_count_deltas`,
`site_visit.revised_site_visit_date`, `questions.revised_due_at`, and
`commercial.bonding_requirement` are `NOT_STATED` because the complete
supplied source set does not state those requested narrative facts.
Do not correct the stated `2025` site-visit registration year; preserve it and
flag the conflict with the 2026 solicitation chronology.
For the synthetic monitoring decision, use `2026-07-08T00:00:00Z`. Mark
`URGENT_REVIEW` when the current response window is open, its due calendar date
is within three calendar dates of the as-of date, and a deadline or scope change
is present. The permitted change categories are `DEADLINE`, `SCOPE`,
`DOCUMENTS`, and `ACKNOWLEDGMENT`.
Every field named in the evidence requirements must cite one complete URL from
the supplied list whose frozen content directly supports that field, together
with a short literal quote found in that same source. Several fields are
supported by more than one supplied source; any reviewed supporting source is
valid. Preserve each URL exactly, including its query string: do not decode,
rewrite, shorten, append to, or otherwise modify it. Quote matching may
normalize whitespace only. In particular, the notice page and the three
reviewed notice-data versions are valid alternatives for
`commercial.funds_currently_available` when the cited source contains the
literal supporting sentence. Do not download attachments, acknowledge an
amendment, submit an offer, register as a bidder, send an email, sign in, click,
type, press a key, or take any state-changing action. Navigate directly to each
complete supplied URL.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
REQUIRED OUTPUT JSON SCHEMA:
{
"$defs": {
"citation": {
"additionalProperties": false,
"properties": {
"quote": {
"minLength": 1,
"type": "string"
},
"url": {
"format": "uri",
"type": "string"
}
},
"required": [
"url",
"quote"
],
"type": "object"
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema",
"items": {
"additionalProperties": false,
"properties": {
"classification": {
"additionalProperties": false,
"properties": {
"naics_code": {
"type": "string"
},
"set_aside": {
"enum": [
"TOTAL_SMALL_BUSINESS",
"UNRESTRICTED",
"OTHER",
"NOT_STATED"
]
}
},
"required": [
"naics_code",
"set_aside"
],
"type": "object"
},
"commercial": {
"additionalProperties": false,
"properties": {
"award_amount": {
"description": "Use NOT_FOUND for this frozen case because the concrete award value is absent from the official data structures.",
"type": [
"string",
"number"
]
},
"bonding_requirement": {
"description": "Use NOT_STATED for this frozen case.",
"type": "string"
},
"funds_currently_available": {
"description": "Whether funds are presently available; the exact notice-page statement is valid evidence.",
"type": "boolean"
}
},
"required": [
"award_amount",
"bonding_requirement",
"funds_currently_available"
],
"type": "object"
},
"document_control": {
"additionalProperties": false,
"properties": {
"amendment_filenames": {
"description": "Canonical source filenames oldest-to-newest. Only runs of whitespace are comparison-normalized; spelling and punctuation are significant.",
"items": {
"type": "string"
},
"type": "array"
},
"attachment_count": {
"minimum": 0,
"type": "integer"
},
"latest_sow_filename": {
"description": "Canonical exact source filename. Only runs of whitespace are comparison-normalized; spelling and punctuation are significant.",
"type": "string"
},
"sow_filenames": {
"description": "Canonical source filenames oldest-to-newest. Only runs of whitespace are comparison-normalized; spelling and punctuation are significant.",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"attachment_count",
"amendment_filenames",
"sow_filenames",
"latest_sow_filename"
],
"type": "object"
},
"evidence": {
"additionalProperties": {
"$ref": "#/$defs/citation"
},
"type": "object"
},
"identity": {
"additionalProperties": false,
"properties": {
"current_version_id": {
"type": "string"
},
"lifecycle_status": {
"description": "Lifecycle status at monitoring.as_of_at (2026-07-08T00:00:00Z), not the later status when the frozen source was fetched.",
"enum": [
"ACTIVE",
"ARCHIVED",
"CANCELLED"
]
},
"solicitation_number": {
"type": "string"
},
"title": {
"type": "string"
}
},
"required": [
"title",
"solicitation_number",
"current_version_id",
"lifecycle_status"
],
"type": "object"
},
"monitoring": {
"additionalProperties": false,
"properties": {
"as_of_at": {
"format": "date-time",
"type": "string"
},
"change_categories": {
"items": {
"enum": [
"DEADLINE",
"SCOPE",
"DOCUMENTS",
"ACKNOWLEDGMENT"
]
},
"type": "array"
},
"current_due_date_open": {
"type": "boolean"
},
"deadline_extension_preserved_response_window": {
"type": "boolean"
},
"prior_due_date_passed": {
"type": "boolean"
},
"review_priority": {
"enum": [
"URGENT_REVIEW",
"NORMAL_REVIEW",
"NO_ACTION"
]
}
},
"required": [
"as_of_at",
"prior_due_date_passed",
"current_due_date_open",
"deadline_extension_preserved_response_window",
"change_categories",
"review_priority"
],
"type": "object"
},
"official_urls": {
"additionalProperties": false,
"properties": {
"after_amendment_0001_version": {
"format": "uri",
"type": "string"
},
"attachment_index": {
"format": "uri",
"type": "string"
},
"baseline_version": {
"format": "uri",
"type": "string"
},
"current_notice_data": {
"format": "uri",
"type": "string"
},
"notice": {
"format": "uri",
"type": "string"
}
},
"required": [
"notice",
"current_notice_data",
"baseline_version",
"after_amendment_0001_version",
"attachment_index"
],
"type": "object"
},
"place": {
"additionalProperties": false,
"properties": {
"city": {
"type": "string"
},
"postal_code": {
"type": "string"
},
"state": {
"description": "Place-of-performance state. WI and Wisconsin are equivalent.",
"type": "string"
}
},
"required": [
"city",
"state",
"postal_code"
],
"type": "object"
},
"questions": {
"additionalProperties": false,
"properties": {
"original_due_at": {
"format": "date-time",
"type": "string"
},
"revised_due_at": {
"description": "Use NOT_STATED for this frozen case.",
"type": "string"
}
},
"required": [
"original_due_at",
"revised_due_at"
],
"type": "object"
},
"scope_changes": {
"additionalProperties": false,
"properties": {
"amendment_0001_specific_scope_changes": {
"description": "Use NOT_STATED for this frozen case; the supplied history does not enumerate the specific Amendment 0001 changes.",
"type": "string"
},
"amendment_0001_updated_sow_published": {
"type": "boolean"
},
"amendment_0002_changed_project_areas": {
"items": {
"type": "integer"
},
"type": "array"
},
"amendment_0002_exact_window_count_deltas": {
"description": "Use NOT_STATED for this frozen case; the supplied history names areas but gives no numeric count deltas.",
"type": "string"
},
"amendment_0002_updated_sow_published": {
"type": "boolean"
}
},
"required": [
"amendment_0001_updated_sow_published",
"amendment_0001_specific_scope_changes",
"amendment_0002_changed_project_areas",
"amendment_0002_exact_window_count_deltas",
"amendment_0002_updated_sow_published"
],
"type": "object"
},
"site_visit": {
"additionalProperties": false,
"properties": {
"location": {
"type": "string"
},
"registration_deadline_as_stated": {
"type": "string"
},
"registration_year_conflict": {
"type": "boolean"
},
"revised_site_visit_date": {
"description": "Use NOT_STATED for this frozen case.",
"type": "string"
},
"scheduled_date_as_stated": {
"type": "string"
},
"scheduled_time_as_stated": {
"type": "string"
}
},
"required": [
"scheduled_date_as_stated",
"scheduled_time_as_stated",
"location",
"registration_deadline_as_stated",
"registration_year_conflict",
"revised_site_visit_date"
],
"type": "object"
},
"timeline": {
"additionalProperties": false,
"properties": {
"after_amendment_0001_due_at": {
"format": "date-time",
"type": "string"
},
"amendment_0001_posted_at": {
"format": "date-time",
"type": "string"
},
"amendment_0001_version_id": {
"type": "string"
},
"amendments_requiring_acknowledgment": {
"description": "Oldest-to-newest amendment identifiers. Bare 0001/0002 and corresponding Amendment-prefixed forms are equivalent; order remains significant.",
"items": {
"type": "string"
},
"type": "array"
},
"baseline_due_at": {
"format": "date-time",
"type": "string"
},
"baseline_posted_at": {
"format": "date-time",
"type": "string"
},
"baseline_version_id": {
"type": "string"
},
"current_due_at": {
"format": "date-time",
"type": "string"
},
"current_posted_at": {
"format": "date-time",
"type": "string"
},
"deadline_changes_count": {
"minimum": 0,
"type": "integer"
},
"deadline_sequence": {
"items": {
"format": "date",
"type": "string"
},
"type": "array"
},
"total_extension_calendar_days": {
"type": "integer"
}
},
"required": [
"baseline_version_id",
"baseline_posted_at",
"baseline_due_at",
"amendment_0001_version_id",
"amendment_0001_posted_at",
"after_amendment_0001_due_at",
"current_posted_at",
"current_due_at",
"deadline_sequence",
"deadline_changes_count",
"total_extension_calendar_days",
"amendments_requiring_acknowledgment"
],
"type": "object"
}
},
"required": [
"identity",
"classification",
"place",
"timeline",
"scope_changes",
"site_visit",
"questions",
"document_control",
"commercial",
"monitoring",
"official_urls",
"evidence"
],
"type": "object"
},
"maxItems": 1,
"minItems": 1,
"type": "array"
}
FROZEN TASK INPUT:
# Frozen task input: construction addenda and deadline-change monitoring
Reconcile SAM.gov solicitation `W50S9H26QA018`, *Volk Field Billeting Blind
Replacement – Multiple Buildings*, as of `2026-07-08T00:00:00Z`.
Use only these official public sources:
- Notice page: https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view
- Current notice data: https://sam.gov/api/prod/opps/v2/opportunities/dfc5bd2c65ae4a4bb910a395175a3343?latest=true
- Baseline before Amendment 0001: https://sam.gov/api/prod/opps/v2/opportunities/70d3a1854d2f445a83e9bcbdcda10221
- State after Amendment 0001: https://sam.gov/api/prod/opps/v2/opportunities/50bcb3f873654600a10619194aba8640
- Current attachment index: https://sam.gov/api/prod/opps/v3/opportunities/dfc5bd2c65ae4a4bb910a395175a3343/resources
Return every field defined by `output.schema.json`. Evidence is a top-level
object keyed by exact dotted field path. Provide one exact-URL quote for:
- `identity.solicitation_number`
- `timeline.baseline_due_at`
- `timeline.after_amendment_0001_due_at`
- `timeline.current_due_at`
- `timeline.amendments_requiring_acknowledgment`
- `scope_changes.amendment_0002_changed_project_areas`
- `site_visit.registration_deadline_as_stated`
- `questions.original_due_at`
- `document_control.latest_sow_filename`
- `commercial.funds_currently_available`
Each evidence value must be `{"url": "...", "quote": "..."}`. Cite any exact
URL above whose frozen content directly supports the complete field value and
contains the literal quote. Preserve the URL exactly, including its query
string; do not decode, rewrite, shorten, append to, or otherwise modify it.
Whitespace normalization is allowed when matching quotes. For
`commercial.funds_currently_available`, the notice page, current notice data,
baseline version, and state after Amendment 0001 are the reviewed alternatives;
no other source is accepted for that field.
Output conventions for this frozen case:
- `place.state` may be `WI` or `Wisconsin`.
- `identity.lifecycle_status` is the status at `2026-07-08T00:00:00Z`, not the
later status when the source snapshot was fetched. The response deadline was
still open, the notice was not cancelled, and automatic archival was dated
July 25, so use `ACTIVE`.
- Keep amendment acknowledgments oldest-to-newest. Bare IDs `0001`, `0002` are
canonical; the corresponding `Amendment 0001`, `Amendment 0002` forms are
equivalent, but reversed order is not.
- Preserve filename spelling, punctuation, and order. Only runs of whitespace
inside filename fields are treated as equivalent.
- Use `NOT_FOUND` only for `commercial.award_amount`.
- Use `NOT_STATED` for
`scope_changes.amendment_0001_specific_scope_changes`,
`scope_changes.amendment_0002_exact_window_count_deltas`,
`site_visit.revised_site_visit_date`, `questions.revised_due_at`, and
`commercial.bonding_requirement`. Do not swap the two missing-value tokens.
Expected output records — evaluator only
[
{
"identity": {
"title": "VOLK FIELD BILLETING BLIND REPLACEMENT - MULTIPLE BUILDINGS",
"solicitation_number": "W50S9H26QA018",
"current_version_id": "dfc5bd2c65ae4a4bb910a395175a3343",
"lifecycle_status": "ACTIVE"
},
"classification": {
"naics_code": "337920",
"set_aside": "TOTAL_SMALL_BUSINESS"
},
"place": {
"city": "Camp Douglas",
"state": "WI",
"postal_code": "54618"
},
"timeline": {
"baseline_version_id": "70d3a1854d2f445a83e9bcbdcda10221",
"baseline_posted_at": "2026-06-25T16:07:33.058Z",
"baseline_due_at": "2026-06-29T13:00:00-05:00",
"amendment_0001_version_id": "50bcb3f873654600a10619194aba8640",
"amendment_0001_posted_at": "2026-06-26T13:26:26.050Z",
"after_amendment_0001_due_at": "2026-07-06T13:00:00-05:00",
"current_posted_at": "2026-07-02T14:40:49.843Z",
"current_due_at": "2026-07-10T13:00:00-05:00",
"deadline_sequence": [
"2026-06-29",
"2026-07-06",
"2026-07-10"
],
"deadline_changes_count": 2,
"total_extension_calendar_days": 11,
"amendments_requiring_acknowledgment": [
"0001",
"0002"
]
},
"scope_changes": {
"amendment_0001_updated_sow_published": true,
"amendment_0001_specific_scope_changes": "NOT_STATED",
"amendment_0002_changed_project_areas": [
2,
5,
6,
7
],
"amendment_0002_exact_window_count_deltas": "NOT_STATED",
"amendment_0002_updated_sow_published": true
},
"site_visit": {
"scheduled_date_as_stated": "16 JUNE",
"scheduled_time_as_stated": "9:00 AM CDT",
"location": "100 Independence Dr. Camp Douglas, WI 54618, BLDG 475",
"registration_deadline_as_stated": "9 JUNE 2025 at 11:00 AM CDT",
"registration_year_conflict": true,
"revised_site_visit_date": "NOT_STATED"
},
"questions": {
"original_due_at": "2026-06-22T13:00:00-05:00",
"revised_due_at": "NOT_STATED"
},
"document_control": {
"attachment_count": 10,
"amendment_filenames": [
"SOLICITATION AMENDMENT 0001 - W50S9H26QA0180001 SF 30.pdf",
"SOLICITATION AMENDMENT 0002 - W50S9H26QA0180002 SF 30.pdf"
],
"sow_filenames": [
"STATEMENT OF WORK - VOLK FIELD BILLETING BLIND REPLACEMENT - MULTIPLE BUILDINGS.pdf",
"STATEMENT OF WORK - LODGING FACILITY BLIND REPLACEMENT - UPDATED 6.25.2026.pdf",
"STATEMENT OF WORK - LODGING FACILITY BLIND REPLACEMENT - UPDATED 7.2.2026.pdf"
],
"latest_sow_filename": "STATEMENT OF WORK - LODGING FACILITY BLIND REPLACEMENT - UPDATED 7.2.2026.pdf"
},
"commercial": {
"award_amount": "NOT_FOUND",
"bonding_requirement": "NOT_STATED",
"funds_currently_available": false
},
"monitoring": {
"as_of_at": "2026-07-08T00:00:00Z",
"prior_due_date_passed": true,
"current_due_date_open": true,
"deadline_extension_preserved_response_window": true,
"change_categories": [
"DEADLINE",
"SCOPE",
"DOCUMENTS",
"ACKNOWLEDGMENT"
],
"review_priority": "URGENT_REVIEW"
},
"official_urls": {
"notice": "https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view",
"current_notice_data": "https://sam.gov/api/prod/opps/v2/opportunities/dfc5bd2c65ae4a4bb910a395175a3343?latest=true",
"baseline_version": "https://sam.gov/api/prod/opps/v2/opportunities/70d3a1854d2f445a83e9bcbdcda10221",
"after_amendment_0001_version": "https://sam.gov/api/prod/opps/v2/opportunities/50bcb3f873654600a10619194aba8640",
"attachment_index": "https://sam.gov/api/prod/opps/v3/opportunities/dfc5bd2c65ae4a4bb910a395175a3343/resources"
}
}
]Field comparison and evidence rules
{
"record_key": "identity.solicitation_number",
"success_threshold": "0.90",
"evidence_threshold": "1.00",
"field_rules": [
{
"name": "identity.title",
"kind": "exact"
},
{
"name": "identity.solicitation_number",
"kind": "exact",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data",
"baseline",
"amendment_0001"
]
},
{
"name": "identity.current_version_id",
"kind": "exact"
},
{
"name": "identity.lifecycle_status",
"kind": "enum",
"enum_values": [
"ACTIVE",
"ARCHIVED",
"CANCELLED"
]
},
{
"name": "classification.naics_code",
"kind": "exact"
},
{
"name": "classification.set_aside",
"kind": "enum",
"enum_values": [
"TOTAL_SMALL_BUSINESS",
"UNRESTRICTED",
"OTHER",
"NOT_STATED"
]
},
{
"name": "place.city",
"kind": "exact"
},
{
"name": "place.state",
"kind": "aliases",
"aliases": {
"Wisconsin": "WI"
}
},
{
"name": "place.postal_code",
"kind": "exact"
},
{
"name": "timeline.baseline_version_id",
"kind": "exact"
},
{
"name": "timeline.baseline_posted_at",
"kind": "datetime"
},
{
"name": "timeline.baseline_due_at",
"kind": "datetime",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data",
"baseline",
"amendment_0001"
]
},
{
"name": "timeline.amendment_0001_version_id",
"kind": "exact"
},
{
"name": "timeline.amendment_0001_posted_at",
"kind": "datetime"
},
{
"name": "timeline.after_amendment_0001_due_at",
"kind": "datetime",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data",
"amendment_0001"
]
},
{
"name": "timeline.current_posted_at",
"kind": "datetime"
},
{
"name": "timeline.current_due_at",
"kind": "datetime",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data"
]
},
{
"name": "timeline.deadline_sequence",
"kind": "ordered_list",
"element_kind": "date",
"critical": true
},
{
"name": "timeline.deadline_changes_count",
"kind": "integer"
},
{
"name": "timeline.total_extension_calendar_days",
"kind": "integer",
"critical": true
},
{
"name": "timeline.amendments_requiring_acknowledgment",
"kind": "ordered_list",
"element_kind": "aliases",
"element_aliases": {
"AMENDMENT 0001": "0001",
"Amendment 0001": "0001",
"amendment 0001": "0001",
"AMENDMENT 0002": "0002",
"Amendment 0002": "0002",
"amendment 0002": "0002"
},
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data"
]
},
{
"name": "scope_changes.amendment_0001_updated_sow_published",
"kind": "exact"
},
{
"name": "scope_changes.amendment_0001_specific_scope_changes",
"kind": "exact",
"critical": true
},
{
"name": "scope_changes.amendment_0002_changed_project_areas",
"kind": "ordered_list",
"element_kind": "integer",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data"
]
},
{
"name": "scope_changes.amendment_0002_exact_window_count_deltas",
"kind": "exact",
"critical": true
},
{
"name": "scope_changes.amendment_0002_updated_sow_published",
"kind": "exact"
},
{
"name": "site_visit.scheduled_date_as_stated",
"kind": "exact"
},
{
"name": "site_visit.scheduled_time_as_stated",
"kind": "aliases",
"aliases": {
"9:00 AM CDT": "09:00 CDT",
"09:00 AM CDT": "09:00 CDT"
}
},
{
"name": "site_visit.location",
"kind": "exact"
},
{
"name": "site_visit.registration_deadline_as_stated",
"kind": "exact",
"evidence_sources": [
"notice",
"current_notice_data",
"baseline",
"amendment_0001"
]
},
{
"name": "site_visit.registration_year_conflict",
"kind": "exact",
"critical": true
},
{
"name": "site_visit.revised_site_visit_date",
"kind": "exact",
"critical": true
},
{
"name": "questions.original_due_at",
"kind": "datetime",
"evidence_sources": [
"notice",
"current_notice_data",
"baseline",
"amendment_0001"
]
},
{
"name": "questions.revised_due_at",
"kind": "exact",
"critical": true
},
{
"name": "document_control.attachment_count",
"kind": "integer"
},
{
"name": "document_control.amendment_filenames",
"kind": "ordered_list",
"element_kind": "whitespace_normalized"
},
{
"name": "document_control.sow_filenames",
"kind": "ordered_list",
"element_kind": "whitespace_normalized"
},
{
"name": "document_control.latest_sow_filename",
"kind": "whitespace_normalized",
"critical": true,
"evidence_source": "attachment_index"
},
{
"name": "commercial.award_amount",
"kind": "exact",
"critical": true
},
{
"name": "commercial.bonding_requirement",
"kind": "exact",
"critical": true
},
{
"name": "commercial.funds_currently_available",
"kind": "exact",
"critical": true,
"evidence_sources": [
"notice",
"current_notice_data",
"baseline",
"amendment_0001"
]
},
{
"name": "monitoring.as_of_at",
"kind": "datetime"
},
{
"name": "monitoring.prior_due_date_passed",
"kind": "exact"
},
{
"name": "monitoring.current_due_date_open",
"kind": "exact",
"critical": true
},
{
"name": "monitoring.deadline_extension_preserved_response_window",
"kind": "exact",
"critical": true
},
{
"name": "monitoring.change_categories",
"kind": "unordered_set",
"element_kind": "enum",
"element_enum_values": [
"DEADLINE",
"SCOPE",
"DOCUMENTS",
"ACKNOWLEDGMENT"
],
"critical": true
},
{
"name": "monitoring.review_priority",
"kind": "enum",
"enum_values": [
"URGENT_REVIEW",
"NORMAL_REVIEW",
"NO_ACTION"
],
"critical": true
},
{
"name": "official_urls.notice",
"kind": "exact"
},
{
"name": "official_urls.current_notice_data",
"kind": "exact"
},
{
"name": "official_urls.baseline_version",
"kind": "exact"
},
{
"name": "official_urls.after_amendment_0001_version",
"kind": "exact"
},
{
"name": "official_urls.attachment_index",
"kind": "exact"
}
],
"evidence_sources": {
"notice": {
"url": "https://sam.gov/opp/dfc5bd2c65ae4a4bb910a395175a3343/view",
"path": "fixtures/sam-notice-visible.txt",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
},
"current_notice_data": {
"url": "https://sam.gov/api/prod/opps/v2/opportunities/dfc5bd2c65ae4a4bb910a395175a3343?latest=true",
"path": "fixtures/sam-notice.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
},
"baseline": {
"url": "https://sam.gov/api/prod/opps/v2/opportunities/70d3a1854d2f445a83e9bcbdcda10221",
"path": "fixtures/sam-version-original.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
},
"amendment_0001": {
"url": "https://sam.gov/api/prod/opps/v2/opportunities/50bcb3f873654600a10619194aba8640",
"path": "fixtures/sam-version-amendment-0001.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
},
"attachment_index": {
"url": "https://sam.gov/api/prod/opps/v3/opportunities/dfc5bd2c65ae4a4bb910a395175a3343/resources",
"path": "fixtures/sam-resources.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
}
}
}Bid intake
Construction opportunity intake and surrogate qualification. Pack 6; reviewed scoring pack.
Executed by: Claude Fable 5.1, Claude Opus 5, Claude Sonnet 5, GPT-5.6 Sol, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.6, Kimi K3.
Time limit: 10 minutes.
Supplied source URLs
Task prompt template
Review the supplied archived construction opportunity using only the supplied official public SAM.gov URLs and the Kernel browser. Return exactly one record matching the supplied output contract. This is a synthetic qualification exercise that resembles construction business-development intake without using a customer's criteria or data. Apply this frozen surrogate filter: - service states: California or Texas; - accepted primary NAICS codes: `236220` or `237310`; - accepted scope: commercial-building or civil/site construction; - a total-small-business set-aside is acceptable; and - the opportunity must still be open at `2026-08-24T00:00:00Z` to receive an `ACTIONABLE_NOW` verdict. Return `ACTIONABLE_NOW` if every condition passes, `RELEVANT_BUT_CLOSED` if the geography, NAICS, scope, and set-aside pass but the opportunity is no longer open, and `NOT_RELEVANT` otherwise. Preserve the component booleans used to reach the verdict. Use the exact `postedDate` timestamp from the notice-data URL for `dates.published_at`; do not substitute the minute-rounded timestamp displayed on the notice page. Either the official state code or the official full state name is acceptable for `place.state`. The abstention tokens are deliberately distinct. Use `NOT_FOUND` for the concrete structured values `classification.secondary_naics_code` and `scope.award_amount`, because those entries are absent from the complete supplied records. Use `NOT_STATED` for `scope.site_visit_date`, because the otherwise descriptive notice never states a site-visit date. Do not invent any of these values. Every field named in the evidence requirements must cite an exact supplied URL and a short quote that is present in that URL's frozen content. For solicitation number, offers-due time, NAICS, state, work type, and begin-after-award, either the notice page or notice-data URL is accepted because both frozen sources directly support those fields. For primary email, use the notice-data URL. For `attachments.names`, return four distinct citations to the attachment-index URL, one quoting each filename. Modified URLs, unsupported paraphrases, and quotes not present in the cited source fail evidence validation. Preserve filename text and order; only insignificant leading, trailing, or repeated whitespace is ignored by scoring. Do not submit an offer, register as a bidder, send an email, sign in, download an attachment, click, type, press a key, or take any state-changing action. The supplied URLs are complete; navigate to each directly. Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and `BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
Reconstructed runner prompt and output contract
BENCHMARK EXECUTION CONTROL (mandatory):
- Use browser tools for the browser work in this task.
- Every browser_start_session call must explicitly pass provider="kernel".
- Start with the supplied reserved ephemeral site_key and pass
clear_cookies=false exactly. The run plan generated this never-before-issued
profile key so the browser starts empty without triggering asynchronous
deletion of another measured attempt's profile.
- Call browser_start_session exactly once. If that start fails, report the
benchmark error and stop instead of retrying with the same or another key.
- Never omit provider and never request Browserbase or automatic fallback.
- Inspect each browser session result. If it identifies any provider other than
Kernel, stop immediately and return a benchmark error instead of continuing.
- When the browser work is complete, call browser_close with neither session_id
nor close_all so the current chat's session is released without an approval.
- Do not take any state-changing action except the task card's explicitly
allowed benchmark-owned Save draft action.
TASK-SPECIFIC BROWSER IDENTITY (mandatory):
- Every browser_start_session call must pass site_key="benchmark-profile-placeholder" exactly. It is an ephemeral profile identifier frozen in this run plan; do not turn it into a URL or Markdown link.
- Every browser_start_session call must pass clear_cookies=false exactly.
- The only permitted tool names for this task are: list_tools, browser_start_session, browser_navigate, browser_snapshot, browser_scroll, browser_open_tab, browser_switch_tab, browser_close_tab, browser_close.
- The measured tool-invocation counts must satisfy: browser_close: exactly 1; browser_start_session: exactly 1.
- Every browser_start_session call must pass url="https://sam.gov/opp/9998b42f9e27421caa565fead7d8d416/view" exactly. This is the canonical start URL for the task.
- URL-bearing browser calls may use only these exact resource URLs: https://sam.gov/opp/9998b42f9e27421caa565fead7d8d416/view; https://sam.gov/api/prod/opps/v2/opportunities/9998b42f9e27421caa565fead7d8d416?latest=true; https://sam.gov/api/prod/opps/v3/opportunities/9998b42f9e27421caa565fead7d8d416/resources. A fragment-only page or anchor selector is allowed on a listed resource; do not alter its scheme, host, path, or query.
- You must visit every listed URL using at least one URL-bearing browser call; opening only a subset fails the measured run.
Review the supplied archived construction opportunity using only the supplied
official public SAM.gov URLs and the Kernel browser. Return exactly one record
matching the supplied output contract.
This is a synthetic qualification exercise that resembles construction
business-development intake without using a customer's criteria or data. Apply
this frozen surrogate filter:
- service states: California or Texas;
- accepted primary NAICS codes: `236220` or `237310`;
- accepted scope: commercial-building or civil/site construction;
- a total-small-business set-aside is acceptable; and
- the opportunity must still be open at `2026-08-24T00:00:00Z` to receive an
`ACTIONABLE_NOW` verdict.
Return `ACTIONABLE_NOW` if every condition passes, `RELEVANT_BUT_CLOSED` if the
geography, NAICS, scope, and set-aside pass but the opportunity is no longer
open, and `NOT_RELEVANT` otherwise. Preserve the component booleans used to
reach the verdict.
Use the exact `postedDate` timestamp from the notice-data URL for
`dates.published_at`; do not substitute the minute-rounded timestamp displayed
on the notice page. Either the official state code or the official full state
name is acceptable for `place.state`.
The abstention tokens are deliberately distinct. Use `NOT_FOUND` for the
concrete structured values `classification.secondary_naics_code` and
`scope.award_amount`, because those entries are absent from the complete
supplied records. Use `NOT_STATED` for `scope.site_visit_date`, because the
otherwise descriptive notice never states a site-visit date. Do not invent any
of these values.
Every field named in the evidence requirements must cite an exact supplied URL
and a short quote that is present in that URL's frozen content. For solicitation
number, offers-due time, NAICS, state, work type, and begin-after-award, either
the notice page or notice-data URL is accepted because both frozen sources
directly support those fields. For primary email, use the notice-data URL. For
`attachments.names`, return four distinct citations to the attachment-index
URL, one quoting each filename. Modified URLs, unsupported paraphrases, and
quotes not present in the cited source fail evidence validation. Preserve
filename text and order; only insignificant leading, trailing, or repeated
whitespace is ignored by scoring.
Do not submit an offer, register as a bidder, send an email, sign in, download
an attachment, click, type, press a key, or take any state-changing action. The
supplied URLs are complete; navigate to each directly.
Emit only a JSON array containing one record between `BENCHMARK_JSON_BEGIN` and
`BENCHMARK_JSON_END`. Do not wrap the JSON in a Markdown code fence.
REQUIRED OUTPUT JSON SCHEMA:
{
"$defs": {
"citation": {
"additionalProperties": false,
"properties": {
"quote": {
"minLength": 1,
"type": "string"
},
"url": {
"format": "uri",
"type": "string"
}
},
"required": [
"url",
"quote"
],
"type": "object"
}
},
"$schema": "https://json-schema.org/draft/2020-12/schema",
"items": {
"additionalProperties": false,
"properties": {
"attachments": {
"additionalProperties": false,
"properties": {
"count": {
"minimum": 0,
"type": "integer"
},
"names": {
"items": {
"description": "Official filename; insignificant whitespace differences are ignored.",
"type": "string"
},
"type": "array"
}
},
"required": [
"count",
"names"
],
"type": "object"
},
"buyer": {
"additionalProperties": false,
"properties": {
"department": {
"type": "string"
},
"office": {
"type": "string"
},
"sub_tier": {
"type": "string"
}
},
"required": [
"department",
"sub_tier",
"office"
],
"type": "object"
},
"classification": {
"additionalProperties": false,
"properties": {
"naics_code": {
"type": "string"
},
"psc_code": {
"type": "string"
},
"secondary_naics_code": {
"description": "Use NOT_FOUND because the complete supplied records contain no secondary NAICS code.",
"type": "string"
},
"set_aside": {
"enum": [
"TOTAL_SMALL_BUSINESS",
"UNRESTRICTED",
"OTHER",
"NOT_STATED"
]
}
},
"required": [
"naics_code",
"secondary_naics_code",
"psc_code",
"set_aside"
],
"type": "object"
},
"contact": {
"additionalProperties": false,
"properties": {
"primary_email": {
"type": "string"
},
"primary_name": {
"type": "string"
}
},
"required": [
"primary_name",
"primary_email"
],
"type": "object"
},
"dates": {
"additionalProperties": false,
"properties": {
"inactive_date": {
"format": "date",
"type": "string"
},
"offers_due_at": {
"format": "date-time",
"type": "string"
},
"published_at": {
"description": "Precise postedDate from the official notice data; timezone-equivalent ISO 8601 representations are accepted.",
"format": "date-time",
"type": "string"
}
},
"required": [
"published_at",
"offers_due_at",
"inactive_date"
],
"type": "object"
},
"evidence": {
"additionalProperties": {
"oneOf": [
{
"$ref": "#/$defs/citation"
},
{
"items": {
"$ref": "#/$defs/citation"
},
"type": "array"
}
]
},
"type": "object"
},
"identity": {
"additionalProperties": false,
"properties": {
"lifecycle_status": {
"enum": [
"ACTIVE",
"ARCHIVED",
"CANCELLED"
]
},
"opportunity_id": {
"type": "string"
},
"solicitation_number": {
"type": "string"
},
"title": {
"type": "string"
}
},
"required": [
"title",
"solicitation_number",
"opportunity_id",
"lifecycle_status"
],
"type": "object"
},
"official_urls": {
"additionalProperties": false,
"properties": {
"attachment_index": {
"format": "uri",
"type": "string"
},
"notice": {
"format": "uri",
"type": "string"
},
"notice_data": {
"format": "uri",
"type": "string"
}
},
"required": [
"notice",
"notice_data",
"attachment_index"
],
"type": "object"
},
"performance": {
"additionalProperties": false,
"properties": {
"begin_within_calendar_days_after_award": {
"minimum": 0,
"type": "integer"
},
"complete_within_calendar_days": {
"minimum": 0,
"type": "integer"
}
},
"required": [
"begin_within_calendar_days_after_award",
"complete_within_calendar_days"
],
"type": "object"
},
"place": {
"additionalProperties": false,
"properties": {
"city": {
"type": "string"
},
"postal_code": {
"type": "string"
},
"state": {
"description": "Official state code or official full state name.",
"type": "string"
}
},
"required": [
"city",
"state",
"postal_code"
],
"type": "object"
},
"qualification": {
"additionalProperties": false,
"properties": {
"as_of_at": {
"format": "date-time",
"type": "string"
},
"geography_match": {
"type": "boolean"
},
"naics_match": {
"type": "boolean"
},
"scope_match": {
"type": "boolean"
},
"set_aside_acceptable": {
"type": "boolean"
},
"status_open": {
"type": "boolean"
},
"verdict": {
"enum": [
"ACTIONABLE_NOW",
"RELEVANT_BUT_CLOSED",
"NOT_RELEVANT"
]
}
},
"required": [
"as_of_at",
"geography_match",
"naics_match",
"scope_match",
"set_aside_acceptable",
"status_open",
"verdict"
],
"type": "object"
},
"scope": {
"additionalProperties": false,
"properties": {
"award_amount": {
"description": "Use NOT_FOUND because the complete supplied records contain no award amount.",
"type": [
"string",
"number"
]
},
"site_visit_date": {
"description": "Use NOT_STATED because the descriptive notice does not state a site-visit date.",
"type": "string"
},
"work_type": {
"enum": [
"COMMERCIAL_BUILDING_CONSTRUCTION",
"CIVIL_SITE_CONSTRUCTION",
"OTHER",
"NOT_STATED"
]
}
},
"required": [
"work_type",
"award_amount",
"site_visit_date"
],
"type": "object"
}
},
"required": [
"identity",
"buyer",
"dates",
"classification",
"place",
"scope",
"performance",
"attachments",
"contact",
"qualification",
"official_urls",
"evidence"
],
"type": "object"
},
"maxItems": 1,
"minItems": 1,
"type": "array"
}
FROZEN TASK INPUT:
# Frozen task input: construction opportunity intake and qualification
Research archived SAM.gov solicitation `70Z03126RALAM2004`, *Install Gravel
Pad*, as of `2026-08-24T00:00:00Z`.
Use only these official public sources:
- Notice page: https://sam.gov/opp/9998b42f9e27421caa565fead7d8d416/view
- Notice data: https://sam.gov/api/prod/opps/v2/opportunities/9998b42f9e27421caa565fead7d8d416?latest=true
- Attachment index: https://sam.gov/api/prod/opps/v3/opportunities/9998b42f9e27421caa565fead7d8d416/resources
Return every field defined by `output.schema.json`. Evidence is a top-level
object keyed by exact dotted field path. Provide evidence for:
- `identity.solicitation_number`
- `dates.offers_due_at`
- `classification.naics_code`
- `place.state`
- `scope.work_type`
- `performance.begin_within_calendar_days_after_award`
- `contact.primary_email`
- `attachments.names`
Each ordinary evidence value must be `{"url": "...", "quote": "..."}`.
Solicitation number, offers-due time, NAICS, state, work type, and
begin-after-award may cite either the exact notice-page URL or exact notice-data
URL when the cited quote appears in that source. Primary email must cite the
notice-data URL. `attachments.names` must contain an array of four such objects
using the attachment-index URL, one distinct filename quote per attachment.
Expected output records — evaluator only
[
{
"identity": {
"title": "Install Gravel Pad",
"solicitation_number": "70Z03126RALAM2004",
"opportunity_id": "9998b42f9e27421caa565fead7d8d416",
"lifecycle_status": "ARCHIVED"
},
"buyer": {
"department": "HOMELAND SECURITY, DEPARTMENT OF",
"sub_tier": "US COAST GUARD",
"office": "BASE ALAMEDA(00031)"
},
"dates": {
"published_at": "2025-11-15T02:57:52.270+00:00",
"offers_due_at": "2025-11-21T12:00:00-06:00",
"inactive_date": "2025-12-06"
},
"classification": {
"naics_code": "237310",
"secondary_naics_code": "NOT_FOUND",
"psc_code": "Y1LB",
"set_aside": "TOTAL_SMALL_BUSINESS"
},
"place": {
"city": "Weslaco",
"state": "TX",
"postal_code": "78596"
},
"scope": {
"work_type": "CIVIL_SITE_CONSTRUCTION",
"award_amount": "NOT_FOUND",
"site_visit_date": "NOT_STATED"
},
"performance": {
"begin_within_calendar_days_after_award": 7,
"complete_within_calendar_days": 10
},
"attachments": {
"count": 4,
"names": [
"70Z03126RALAM2004.pdf",
"Final Design Specifications_Gravel Pad_RV01.pdf",
"Final Design Drawings_Gravel Pad_RV01.pdf",
"WRR TX20250255 Hidalgo 20250314.pdf"
]
},
"contact": {
"primary_name": "Shawn Jenkins",
"primary_email": "[email protected]"
},
"qualification": {
"as_of_at": "2026-08-24T00:00:00Z",
"geography_match": true,
"naics_match": true,
"scope_match": true,
"set_aside_acceptable": true,
"status_open": false,
"verdict": "RELEVANT_BUT_CLOSED"
},
"official_urls": {
"notice": "https://sam.gov/opp/9998b42f9e27421caa565fead7d8d416/view",
"notice_data": "https://sam.gov/api/prod/opps/v2/opportunities/9998b42f9e27421caa565fead7d8d416?latest=true",
"attachment_index": "https://sam.gov/api/prod/opps/v3/opportunities/9998b42f9e27421caa565fead7d8d416/resources"
}
}
]Field comparison and evidence rules
{
"record_key": "identity.solicitation_number",
"success_threshold": "0.90",
"evidence_threshold": "1.00",
"field_rules": [
{
"name": "identity.title",
"kind": "exact"
},
{
"name": "identity.solicitation_number",
"kind": "exact",
"critical": true,
"evidence_sources": [
"notice",
"notice_data"
]
},
{
"name": "identity.opportunity_id",
"kind": "exact"
},
{
"name": "identity.lifecycle_status",
"kind": "enum",
"enum_values": [
"ACTIVE",
"ARCHIVED",
"CANCELLED"
]
},
{
"name": "buyer.department",
"kind": "exact"
},
{
"name": "buyer.sub_tier",
"kind": "exact"
},
{
"name": "buyer.office",
"kind": "exact"
},
{
"name": "dates.published_at",
"kind": "datetime"
},
{
"name": "dates.offers_due_at",
"kind": "datetime",
"critical": true,
"evidence_sources": [
"notice",
"notice_data"
]
},
{
"name": "dates.inactive_date",
"kind": "date"
},
{
"name": "classification.naics_code",
"kind": "exact",
"critical": true,
"evidence_sources": [
"notice",
"notice_data"
]
},
{
"name": "classification.secondary_naics_code",
"kind": "exact",
"critical": true
},
{
"name": "classification.psc_code",
"kind": "exact"
},
{
"name": "classification.set_aside",
"kind": "enum",
"enum_values": [
"TOTAL_SMALL_BUSINESS",
"UNRESTRICTED",
"OTHER",
"NOT_STATED"
],
"critical": true
},
{
"name": "place.city",
"kind": "exact"
},
{
"name": "place.state",
"kind": "aliases",
"aliases": {
"Texas": "TX"
},
"critical": true,
"evidence_sources": [
"notice",
"notice_data"
]
},
{
"name": "place.postal_code",
"kind": "exact"
},
{
"name": "scope.work_type",
"kind": "enum",
"enum_values": [
"COMMERCIAL_BUILDING_CONSTRUCTION",
"CIVIL_SITE_CONSTRUCTION",
"OTHER",
"NOT_STATED"
],
"critical": true,
"evidence_sources": [
"notice",
"notice_data"
]
},
{
"name": "scope.award_amount",
"kind": "exact",
"critical": true
},
{
"name": "scope.site_visit_date",
"kind": "exact",
"critical": true
},
{
"name": "performance.begin_within_calendar_days_after_award",
"kind": "integer",
"critical": true,
"evidence_sources": [
"notice",
"notice_data"
]
},
{
"name": "performance.complete_within_calendar_days",
"kind": "integer",
"critical": true
},
{
"name": "attachments.count",
"kind": "integer"
},
{
"name": "attachments.names",
"kind": "ordered_list",
"element_kind": "whitespace_normalized",
"critical": true,
"evidence_sources": [
"attachment_names"
],
"evidence_min_citations": 4
},
{
"name": "contact.primary_name",
"kind": "exact"
},
{
"name": "contact.primary_email",
"kind": "exact",
"evidence_source": "notice_data"
},
{
"name": "qualification.as_of_at",
"kind": "datetime"
},
{
"name": "qualification.geography_match",
"kind": "exact"
},
{
"name": "qualification.naics_match",
"kind": "exact"
},
{
"name": "qualification.scope_match",
"kind": "exact"
},
{
"name": "qualification.set_aside_acceptable",
"kind": "exact"
},
{
"name": "qualification.status_open",
"kind": "exact"
},
{
"name": "qualification.verdict",
"kind": "enum",
"enum_values": [
"ACTIONABLE_NOW",
"RELEVANT_BUT_CLOSED",
"NOT_RELEVANT"
],
"critical": true
},
{
"name": "official_urls.notice",
"kind": "exact"
},
{
"name": "official_urls.notice_data",
"kind": "exact"
},
{
"name": "official_urls.attachment_index",
"kind": "exact"
}
],
"evidence_sources": {
"notice": {
"url": "https://sam.gov/opp/9998b42f9e27421caa565fead7d8d416/view",
"path": "fixtures/sam-notice-visible.txt",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
},
"notice_data": {
"url": "https://sam.gov/api/prod/opps/v2/opportunities/9998b42f9e27421caa565fead7d8d416?latest=true",
"path": "fixtures/sam-notice.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
},
"attachment_names": {
"url": "https://sam.gov/api/prod/opps/v3/opportunities/9998b42f9e27421caa565fead7d8d416/resources",
"path": "fixtures/sam-resources.json",
"format": "text",
"require_quote": true,
"normalize_quote_whitespace": true
}
}
}