What 240 browser-agent runs tell us about choosing a model
Eight models compared on answer correctness, qualified passes, task time and cost across six browser workflows.
8 models · 6 fixed cases · 5 repetitions each · 240 selected executions · Kernel
Updated September 23, 2026.
Choosing a model for a browser agent is not the same as choosing the most impressive chatbot. The job may be entering a form, monitoring a construction opportunity or checking the terms of a contract. The useful output is a completed, verifiable workflow—not just an answer that sounds right.
We compared eight models in AI Drive across six browser workflows, with five repetitions per case. The resulting 240 selected executions measure qualified passes, end-to-end task time and settled AI Drive credits. All browser work used Kernel; email delivery was not part of the test.
No model led on every measure. Claude Fable 5.1 recorded the most qualified passes at 29/30, followed by Kimi K3 at 28/30. GPT-6 Astra had the highest recorded answer-field match rate at 905/905. Gemini 3.8 Flash had the lowest mean task cost, and GPT-5.6 Sol had the lowest median task time. The charts below compare all eight models on each measure and each workflow.
Since then, AI Drive has replaced GPT-5.6 Sol with GPT-6 Sol, Claude Opus 5 with Claude Opus 5.5, and Grok 4.6 with Grok 4.7. The newer models were not tested, and these results should not be read as theirs.
These are observed results on six cases, not a universal ranking. Collection spanned August 27–September 7 across development and production, including Fable through the development ingress; reasoning budgets were not matched, and Kimi was selected from five open-source candidates on this same sample. Details are in the methodology below.
Correct answers and qualified passes
Did every scored answer field match? Did the run meet the complete acceptance checklist? These are different questions. Both charts use the same 30 attempts per model, in alphabetical order.
Every answer field correct
AI DriveRuns where every scored answer field matched the reviewed answer. Citations and execution are assessed separately.
Qualified passes
AI DriveRuns meeting the original answer, evidence, execution and measurement requirements. Some non-critical answer mistakes are allowed.
Neither measure contains the other: a fully correct answer can fail a citation check; a qualified pass can contain a non-critical field error. “Every answer field correct” covers the fields we scored, not every possible claim or untested task. This is not a standalone computer-use ranking.
Dollar estimates use 2,000 AI Drive credits per $1, the historical display basis—not provider API prices or a current pricing quote. The results table shows both units.
What we mean by a qualified pass
A run had to meet the frozen rubric: at least 90% field accuracy, every configured critical field correct, the required source evidence, permitted browser behavior, completion within the time limit and an exact settled-credit receipt. One failed requirement is enough to prevent a qualified pass.
Answer, evidence and execution checks measure different parts of the workflow. We report them separately rather than treating every non-pass as a hallucination.
Why runs did not qualify
Counts below cover the 30 attempts per model. A run may fail more than one rule, so the three failure columns can overlap and must not be added together.
Answer-rule failures mean incomplete output, a wrong critical field, or less than 90% field accuracy. Evidence-rule failures mean a required citation or quotation did not pass. Browser / other failures cover execution, safety, provider verification, timing and measurement requirements.
| Claude Fable 5.1 | 1 | 0 | 0 | 1 |
| Kimi K3 | 2 | 0 | 0 | 2 |
| Claude Opus 5 | 3 | 0 | 0 | 3 |
| GPT-6 Astra | 3 | 0 | 3 | 0 |
| GPT-5.6 Sol | 4 | 2 | 4 | 0 |
| Gemini 3.8 Flash | 5 | 0 | 3 | 2 |
| Claude Sonnet 5 | 10 | 1 | 9 | 4 |
| Grok 4.6 | 11 | 7 | 3 | 2 |
This diagnoses non-passes, not every answer mistake. A passing run can still contain non-critical mistakes. Zero in a failure column does not mean zero factual errors; these are not hallucination rates.
The highest score is a starting point, not the whole decision
Fable's 29/30 was the highest observed qualified-pass count. It passed five of the six cases on every repetition, missing one SEC-contract attempt. Kimi and Claude Opus 5 also had five cases with 5/5 results, despite having lower totals. This is why the total and the per-workflow breakdown belong together.
Kimi's 28/30 is especially relevant to the shortlist: its mean was about 2,064 credits per attempted task, versus 7,949 for Fable. That is a useful observed cost-quality trade-off. The one-pass difference between them is too small, over too few distinct cases, to establish a general reliability advantage for either model.
Mean cost per attempted task
AI DriveAI Drive creditsEstimated dollars · lower is better
Median task time
AI DriveEnd-to-end elapsed time · lower is better
Cost and time tell different stories
Gemini 3.8 Flash averaged 994 credits per attempted task—about $0.50 at the report's illustrative conversion. It was the lowest-cost model among the eight featured cohorts, with 25/30 qualified passes. That is a sensible candidate for monitored, low-consequence work when it meets the quality requirement of the exact task.
GPT-5.6 Sol had the lowest observed median time: 1 minute 53 seconds. Its mean was 2,441 credits, with 26/30 passes. Flash took 3 minutes 26 seconds at the median. If latency matters, the lowest credit bill need not be the best operational choice.
Astra's median was 2 minutes 6 seconds, but its mean task cost was 8,417 credits—approximately $4.21, the highest in this comparison. It passed 27/30. This batch does not support treating the newer model as an automatic replacement for every workflow; the additional spend needs to be justified on the task being deployed.
Qualified passes vs. task cost
AI DriveHigher and farther left is the favorable direction; each point is one model.
Swipe the chart horizontally to read every label.
Read the workflow, not just the headline
The six cases deliberately mix interaction, monitoring and evidence-heavy research. The form is a public demo using synthetic data. Bid intake and addenda trace use construction notices. The legal-oriented cases use public SEC, eCFR and Supreme Court records. No customer credentials, private bid information or live filing were needed.
Each bar below is five executions of one fixed case. A 5/5 result is a repeatability signal on that case—not five independently sampled customers, contracts or regulations. Use it to choose the next validation task, not to skip validation.
Controlled form
AI DriveFill a multi-step demo with synthetic details, verify the confirmation screen and stop before submitting.
Bid intake
AI DriveExtract an archived construction opportunity and apply a fixed, synthetic qualification filter.
Addenda trace
AI DriveTrace solicitation amendments, preserve old and new deadlines, and identify stated scope changes.
SEC contract
AI DriveExtract contract terms from an SEC exhibit and distinguish missing terms from terms that are actually stated.
eCFR regulation
AI DriveTrace a historical regulation change using supplied point-in-time eCFR records.
Court docket
AI DriveReconcile an official court docket and opinion, including dates, recent events and supporting quotations.
Two ways to use these results
High-risk work
When a missed deadline or unsupported contract term carries a large cost, evaluate critical-field errors, unsupported assertions and evidence quality alongside completion rate. Use the per-workflow results to build a shortlist, then validate it on representative tasks from your own operation. Five repetitions of one case are not a deployment guarantee. Retain human approval before consequential action.
Routine checks
For low-consequence, easily checked work, Flash is the observed economy candidate at about 994 credits per task. Use it only where it clears the task's quality threshold, with monitored samples and a route to human review when evidence is missing.
The takeaway
This benchmark exposes trade-offs between answer correctness, qualified passes, speed and cost. Fable had the highest qualified-pass count, Astra the highest answer-field match rate, Flash the lowest mean cost, GPT-5.6 Sol the lowest median time, and Kimi a strong open-source qualified-pass result. Start from the task, decide what failure would cost, and validate the model at that level.
Complete results
Select a column heading to sort. Correct answer fields are matching field instances across all attempts, including failures—not independent facts or a hallucination rate. The adjacent run count requires every scored field to match. Evidence and execution remain part of the separate qualified-pass measure.
Cost is the mean across all 30 attempts; time is the median. “5/5 workflows” counts cases that qualified on every repetition.
“Unsupported fills” counts supplied values where the source-only task required NOT_STATED or NOT_FOUND, out of 105 missing-value field instances. This is not a general hallucination rate: a value can be plausible but unsupported by the specified source. Some non-critical errors still meet the qualified-pass threshold.
| Claude Fable 5.1 | 895/905 (98.9%) | 20/30 | 29/30 | 5/6 | 7,949.4 | $3.97 | 2m 27s | 30/30 | 5/105 | 5/29 |
| Claude Opus 5 | 893/905 (98.7%) | 20/30 | 27/30 | 5/6 | 4,936.2 | $2.47 | 2m 24s | 30/30 | 4/105 | 4/27 |
| Claude Sonnet 5 | 863/905 (95.4%) | 21/30 | 20/30 | 2/6 | 2,112.5 | $1.06 | 3m 00s | 30/30 | 5/105 | 2/20 |
| Gemini 3.8 Flash | 899/905 (99.3%) | 24/30 | 25/30 | 3/6 | 993.5 | $0.50 | 3m 26s | 30/30 | 5/105 | 5/25 |
| GPT-5.6 Sol | 902/905 (99.7%) | 27/30 | 26/30 | 3/6 | 2,441.3 | $1.22 | 1m 53s | 30/30 | 1/105 | 0/26 |
| GPT-6 Astra | 905/905 (100.0%) | 30/30 | 27/30 | 4/6 | 8,417.1 | $4.21 | 2m 06s | 30/30 | 0/105 | 0/27 |
| Grok 4.6 | 897/905 (99.1%) | 22/30 | 19/30 | 2/6 | 2,019.4 | $1.01 | 3m 56s | 30/30 | 1/105 | 0/19 |
| Kimi K3 | 890/905 (98.3%) | 20/30 | 28/30 | 5/6 | 2,064.0 | $1.03 | 3m 11s | 30/30 | 5/105 | 3/28 |
Methodology
We tested eight models across six fixed browser workflows, with five repetitions per workflow, using AI Drive and Kernel. Results include answer correctness, qualified passes, elapsed time and task credits. These are 240 executions of six cases, not 240 different tasks.
Every model's output is evaluated using the same reviewed scoring rules for its workflow. The charts retain the original qualified-pass rubric and all selected attempts. Answer-field and unsupported-fill counts are additional views of those same recorded outputs; costs and times are not recomputed by a new browser run.