What 240 browser-agent runs tell us about choosing a model

What we mean by a qualified pass

A run had to meet the frozen rubric: at least 90% field accuracy, every configured critical field correct, the required source evidence, permitted browser behavior, completion within the time limit and an exact settled-credit receipt. One failed requirement is enough to prevent a qualified pass.

That distinction matters. A wrong deadline, an unsupported quotation and an invalid browser action are different failures. We do not count every non-pass as a hallucination. The tasks also ask the agent to distinguish information that is absent from information that is not stated, rather than fill gaps with plausible guesses.

A qualified pass is not a claim of error-free output

Two examples show what this checklist rewards. Fable qualified on all five docket runs while supplying a majority count of 9 where the reviewed answer expected NOT_STATED. The scorer flags this as an unsupported filled-in value, but the field was non-critical, so those runs could still pass. This does not establish that 9 is false; it failed the task's source-only, no-inference requirement.

Astra matched every scored answer field in all 30 runs, but qualified on 27. One quotation failed the frozen-source check, and two docket responses added page-query parameters to the required citation URLs. These are evidence-contract failures, not proof that the scored factual answers were wrong.

The original rubric and scores remain unchanged for every model. The headline measures completion of the full acceptance checklist—not which model made the fewest factual mistakes. The examples do not establish a different universal winner either.

The highest score is a starting point, not the whole decision

Fable's 29/30 was the highest observed aggregate. It passed five of the six cases on every repetition, missing one SEC-contract attempt. Kimi and Opus also had five cases with 5/5 results, despite having lower totals. This is why the total and the per-workflow breakdown belong together.

Kimi's 28/30 is especially relevant to the shortlist: its mean was about 2,064 credits per attempted task, versus 7,949 for Fable. That is a useful observed cost-quality trade-off. The one-pass difference between them is too small, over too few distinct cases, to establish a general reliability advantage for either model.

The eight-model chart combines earlier cohorts with a fresh production extension. Collection dates, environments and some execution prompts differ; all use the reviewed scoring packs. These comparisons guide further validation, but they do not isolate model capability from the rest of the system.

Cost and time tell different stories

Gemini 3.8 Flash averaged 994 credits per attempted task—about $0.50 at the report's illustrative conversion. It was the lowest-cost model among the eight featured cohorts, with 25/30 qualified passes. That is a sensible candidate for monitored, low-consequence work when it meets the quality requirement of the exact task.

GPT-5.6 Sol had the lowest observed median time: 1 minute 53 seconds. Its mean was 2,441 credits, with 26/30 passes. Flash took 3 minutes 26 seconds at the median. If latency matters, the lowest credit bill need not be the best operational choice.

Astra's median was 2 minutes 6 seconds, but its mean task cost was 8,417 credits—approximately $4.21, the highest in this comparison. It passed 27/30. This batch does not support treating the newer model as an automatic replacement for every workflow; the additional spend needs to be justified on the task being deployed.

Read the workflow, not just the headline

The six cases deliberately mix interaction, monitoring and evidence-heavy research. The form is a public demo using synthetic data. Bid intake and addenda trace use construction notices. The legal-oriented cases use public SEC, eCFR and Supreme Court records. No customer credentials, private bid information or live filing were needed.

Each bar below is five executions of one fixed case. A 5/5 result is a repeatability signal on that case—not five independently sampled customers, contracts or regulations. Use it to choose the next validation task, not to skip validation.

A non-pass does not always mean a wrong answer

Astra's three misses were evidence checks, not failures of the scored factual fields: one quotation did not match the frozen source, and two docket responses added PDF-page query parameters to the required citation URLs. The frozen rubric requires the reviewed URLs exactly. This is a citation-contract failure; it is not proof that the underlying legal answer was false.

Kimi's two misses were browser-execution checks, despite passing its answer and evidence gates. We retained both sets of failures rather than adjusting the rules after seeing which model benefited. The practical lesson is to examine why a run failed, not to reinterpret the qualified-pass rate as a hallucination rate.

The takeaway

The useful outcome of a benchmark is a better deployment decision, not a permanent winner. Fable had the highest observed pass count, Flash the lowest mean cost, Sol the lowest median time, and Kimi a strong open-source result. Start from the task, decide what failure would cost, and validate the model at that level.