Benchmark catalogue

Every's cross-task model reports, including its Senior Engineer benchmark, writing tests, knowledge work, enterprise consulting, and computer-use tasks.

• September 2026 GPT-6 Sol vs. Opus 5.5 comparison across Every's writing, computer-use, and coding tasks.

Model comparison on agentic coding and one-shot reasoning in a multi-agent evaluation suite, with task performance and cost-efficiency results.

Cohere North Micro Vision result report covering document understanding, visual question answering, and other visual-understanding benchmarks.

• Cohere's launch post summarizes the model's visual benchmark positioning.

Vision-language model result post reporting ScreenSpot-v2, RealWorldQA, TextVQA, RefCOCO, and ToolSandbox scores.

• LFM2.5-VL-3B model card and release materials.

Model-configuration guidance comparing coding performance and cost across reasoning-effort levels.

Empirical evaluation reporting that test-driven development harmed coding-agent performance.

Model comparison on self-contained HTML5 physics demos, including token use, cost, and simulation quality.

Harness and model comparison on Databricks' multi-million-line internal codebase.

References ProgramBench, CodeClash, AlgoTune, SWEfficiency, CritPt, and VideoGameBench.

Results across Vals Index, ProgramBench, ProofBench, VibeCodeBench, and Public Benefits Bench.

Comparison of three current model leaderboards.

Model results broken down by production-task category.

Gemini 3 Pro results across Vals benchmarks.

Discussion of tougher coding benchmarks beyond SWE-bench.

Internal evaluation of document and image data-extraction accuracy across 40,000 fields.

Real-world agentic coding comparison across codebases.

High-level software-engineering requests with hidden tests and validation agents.

Production-code quality benchmark measuring whether a maintainer would merge an agent-generated pull request.

Software-engineering-agent benchmark for challenging performance-optimization tasks across real codebases and languages.

• Saved result thread reports Opus 4.5 passing 25% and Gemini 3 Pro moving to second place on GSO.

Resolving real GitHub issues in Python repositories.

Warning OpenAI reports that SWE-bench Verified is increasingly contaminated and recommends SWE-bench Pro instead.

Scale AI's public leaderboard for a harder SWE-bench variant.

• Scale AI's commercial SWE-bench Pro leaderboard.

Ramp's evaluation of background coding agents on financial software-engineering tasks, including task outcomes, work patterns, tool use, cost, and runtime.

Coding-agent benchmark built from private, licensed production codebases, measuring model-and-harness performance on real enterprise engineering tasks.

• Launch announcement for the private-codebase benchmark.

A reproducible software-engineering benchmark.

Continuously refreshed real-world software-engineering tasks.

Efficiency-focused evaluation for software-engineering agents.

• Launch post: 498 optimization tasks across nine data-science, ML, and HPC repositories.

Long-horizon software-engineering benchmark.

Long-horizon whole-repository migration benchmark covering language, framework, platform, and build-toolchain rewrites, graded for migration completeness, behavior, and hidden regressions.

• Launch post reports 20 migrations, 520 runs, and 28 accepted runs (5.4%).
• Official benchmark homepage and leaderboard.
• Paper describing the three-stage audit, behavioral, and agentic-verification protocol.

Interactive coding sessions scored for correctness and user correction.

• Saved methodology thread on measuring correctness and how often users need to correct an agent during interactive coding.

Software-engineering benchmark with cost and performance comparisons between frontier models.

Private-codebase coding-agent evaluation comparing quality, cost, and speed across models and harnesses.

A demanding software-engineering benchmark.

Production software-engineering tasks spanning deployment, integration, and debugging.

Ultra-long-horizon software-engineering benchmark with 20-hour runs; v2 expands the suite to 34 difficult engineering and research tasks.

• FrontierSWE v2 release notes and updated leaderboard.
• Proximal's announcement of the expanded task suite and methodology.
• Independent result post reporting model performance on v2.

Program-level coding tasks and evaluation.

• Opus 5 model-card experiment comparing async and flat multi-agent harnesses with a single-agent baseline.
• Gemini 3.5 and 3.6 Flash benchmark results for binary reimplementation without source, decompilation, or internet access.

Evaluates AI agents that turn prompts into rendered videos by writing code, with a focus on motion-design quality rather than code execution alone.

Long-horizon coding benchmark where agents reimplement complete programs from behavior without their source code.

• August leaderboard update reports Claude Fable 5 solving 64% of tasks and GPT-5.6 Sol 20%.

Codebase-wide engineering evaluation workflow for finding and implementing improvements in performance, quality, dead code, SEO, accessibility, and other maintenance goals.

Iterative coding benchmark that measures correctness, code erosion, and verbosity as agents extend their own solutions across changing requirements.

• Earlier result post introducing the benchmark.
• September 2026 update: GPT-6 Astra entered the benchmark.
• Full-run update covering every scenario for GLM 5.3, GPT-5.6 Sol, and GPT-6 Astra; the author notes the run was resumed after provider outages.

Open coding benchmark on real merged pull requests, combining functional correctness with code quality, security, maintainability, cost, and runtime across reasoning-effort levels.

• September 2026 v4 comparison of GPT-6 Astra and Fable 5.1 with code quality weighted at 33%.
• Open-source harness and reproducible task definitions.

Repository-scale coding benchmark.

Contamination-resistant evaluation of code generation.

Result analysis of coding-agent performance across novel interactive environments and programming languages.

Custom coding evaluation that turns merged pull requests into tasks and measures an agent's resolution rate.

Coding-agent leaderboard with model result updates.

• Result update reporting GPT-5.3-Codex at 86% on IBench.

Coding-agent result comparison covering task performance, cost efficiency, code density, and review quality.

• Grok 4.6 result update on GBENCH's complex multi-agent coding evaluations.

One-shot coding-agent ranking and model comparison.

Agentic coding benchmark and leaderboard.

Scientific-reproducibility tasks that require agents to run and interpret research repositories.

Coding-agent evaluation on realistic React work, including production quality, performance, and accessibility criteria.

Google's Android-specific coding benchmark and leaderboard covering 100 real-world Android development tasks, with quality, latency, and cost reporting.

• Android Developers' update adds new frontier models to the leaderboard.

Game-development agent benchmark result comparing model task-success rates.

Vibe-coding benchmark covering debugging, refactoring, UI, security, cost, and speed.

End-to-end web-app benchmark that assesses whether agents meet product requirements through behavioral test plans.

Next.js's evaluation suite for AI coding agents.

Terminal-agent benchmark and leaderboard for real command-line tasks.

• Vals AI launch of Terminal-Bench 4.0: 66 strict pass/fail terminal tasks scored as avg@3.

Community benchmark for frontier agent work, built by the team behind Terminal-Bench and Harbor.

Compares coding-agent harnesses by speed, cost per solve, and task performance.

• Launch results compare Pi, Cursor, Codex, and OpenCode on coding and Terminal-Bench tasks for speed and cost per solve.

Cognition's model-pairing harness evaluation across DeepSWE, Terminal-Bench, SWE-Atlas QnA, Vals Code Migration, and FrontierCode, focused on task cost and efficiency.

Internal coding-agent evaluation on Databricks' multi-million-line codebase, comparing quality and cost.

Independent task and language-token evaluations of context compression, cache reuse, and multilingual token overhead; some find compression increases total token cost.

• Copilot evaluation: neutral-to-higher token use after compression.
• Hermes evaluation: most Headroom changes increased token cost, with one file-search exception.
• Claude Code real-task benchmark: 17% higher token cost from cache disruption.
• RTK evaluation and analysis of command-output token reduction for coding agents.
• Cross-language token-count comparison finds substantial non-English overhead for some model tokenizers, especially on Hindi and Arabic.

Compares sandbox providers on repeatable developer and CI/CD workloads, including end-to-end workflow time.

• Saved comparison post highlighting sandbox performance, including Run Cloud.

Enterprise-agent benchmark that measures task performance under realistic EDR, SASE, and DLP policy constraints.

Model evaluation on adversarial code reviews, comparing bug findings, token and dollar cost, and latency across reasoning-effort settings.

Code-review agent evaluation based on curated, substantive findings from real merged pull requests.

Open-source comparison of code-review agents, measuring bug detection, code improvements, and developer adoption.

Evaluation of AI code-review performance.

Code-review comparison on 400 real pull requests.

Security-agent benchmark spanning 869 real vulnerabilities across userspace software, V8, and the Linux kernel.

Security-agent benchmark comparing Devin Security Swarm with leading coding agents and analyzers, including Codex and Claude, on 50 real post-training-cutoff vulnerabilities across 14 languages; measures exact-vulnerability recall and cost per scan.

Fresh-CVE security-agent benchmark testing vulnerability rediscovery, recall, consistency, and cost across 32 recent vulnerabilities and three runs per model.

• Aikido's launch post summarizes the 11.7-billion-token evaluation and model results.
• Earlier known-CVE benchmark using 26 vulnerabilities and pass@3 pooling.
• July results thread compares Opus 5 with GPT-5.6 Sol and other models, noting Opus's higher vulnerability recall but unstable results across three runs.

Security-agent benchmark comparison covering accuracy, price-normalized precision, and model cost.

Contamination-resistant software reverse-engineering benchmark with 262 binary instances and 1,572 deterministically graded tasks.

• Vals AI's announcement frames SRE-Bench as a test of understanding binaries without source code.

Agentic GPU-kernel benchmark and leaderboard for writing correct, efficient CUDA kernels.

• KernelBench-Mega result reports a Kimi-Linear decode megakernel at 35.5× the optimized PyTorch baseline on an RTX PRO 6000.
• July result post shows Kimi K3 writing and optimizing kernels for KernelBench.

Cross-provider model comparisons, including coding and speed.

Registry and capability tracker for AI benchmarks, models, and their release history.

Cross-benchmark model rankings with actual cost comparisons, provider and hosting filters, and evidence linked to benchmark results.

Ground-truth vision benchmark across object detection, counting, identification, OCR, data extraction, and image reasoning, with cost and speed reporting.

• Roboflow's earlier GPT-5.6 VLM benchmark report.
• Cost-sorted model comparison from the current Vision Evals leaderboard.
• September 2026 refresh adds Gemini 3.8 Flash and Fable 5.1 to the top 10; Roboflow notes Fable is about 50% faster than the Gemini models ahead of it.
• Vision-model detection deep dive highlighting detail, box precision, OCR, speed, and cost.

Controlled lexical-fingerprint experiment comparing models' use of signature words across identical questions; the linked ox-alpha page analyzes 1,469 complete responses against 374 comparison models.

Frequently refreshed benchmark designed to reduce contamination.

Collection of AI benchmark analyses and datasets.

Cross-benchmark model leaderboard combining quality, cost, speed, context, and confidence data.

Scale AI's general model-evaluation leaderboard.

Kaggle's benchmark hub and leaderboards.

Live, community-voted LLM leaderboard with real-time Vibe Score and Curse Trend metrics.

Practitioner-panel benchmark that compares model releases through engineers' real-world, substantive use.

Aggregate evaluation of model performance across tasks.

Measures whether agents learn from feedback during repeated play.

Continuously rotated, sealed evaluations designed to reduce benchmark gaming and training contamination.

Difficult multimodal reasoning benchmark covering spatial reasoning, visual understanding, and multi-step reasoning across images.

Structured-output benchmark testing JSON and YAML validity, schema adherence, and format instructions through binary pass/fail scoring.

Multimodal spatial-reasoning benchmark covering complex 2D and 3D tasks, with a public model leaderboard and human baseline.

• Original benchmark announcement and SpatialBench description.
• September 2026 update says GPT-6 Astra saturated the spatial-reasoning evaluation.

Post-training-agent benchmark with anti-cheat checks for data contamination, model substitution, API use, and trace lookup.

Adversarial multi-turn debate evaluation with side-swapped matchups and multi-model judging of factual accuracy, rebuttals, and sustained coherence.

• September 2026 result update describing the current Debate Benchmark leaderboard.

Game-playing evaluation comparing a vision-language model with professional GeoGuessr players.

Long-horizon agent learning across scientific, professional, coding, math, and game tasks.

General model-capability benchmark with a public high-score tracker.

Knowledge and hallucination evaluation from Artificial Analysis.

• Gemini 3 Flash result post reports a 91% hallucination rate on AA-Omniscience.

Word-chain benchmark that stress-tests language-model vocabulary, planning, and rule-following.

• Gemini 3 Flash result ranks 12th and notes its lower average word-chain validity.

Measures whether models detect and clearly challenge flawed or misleading premises across domains and techniques.

Reasoning and visual abstraction benchmark comparison.

Game-like evaluation of how well models orient in unfamiliar situations; includes verified frontier-model results.

OpenAI long-context reasoning benchmark for breadth-first graph traversal over very large inputs.

Visual-reasoning benchmark result tracking a new high score and performance by reasoning level.

WeirdML comparison of model performance, result consistency, and cost at a near-saturated frontier score.

Framer's 236-task evaluation of AI agents building and maintaining responsive websites, spanning layout, interactions, and design workflows.

Framer's continuous evaluation of translation models for quality, structural correctness, latency, and cost on real website content.

Multi-step business-automation workflows with cost per task.

Internal production-harness evaluation of agent performance on everyday and long-horizon spreadsheet tasks.

Sales-call coaching benchmark scored against hidden ground truth across synthetic calls.

Ongoing production evaluation of models on anonymized Notion knowledge-work traffic, scored for resolution rate, cost per task, and time; Notion explicitly describes it as a live eval rather than a traditional fixed benchmark.

• Launch post describing the live anonymized-traffic evaluation and ensemble judge.

Hex's frontier benchmark for agentic analytics: 100 realistic Q&A and open-ended data-work tasks in a synthetic warehouse, scored for analytical judgment rather than code generation alone.

• Announcement of DataBench as a benchmark derived from realistic Hex usage rather than pub-trivia questions.

Open benchmark developed with more than 80 licensed mental-health experts to assess safety, context-seeking, user agency, and actionable guidance across realistic conversations from everyday stress to acute crisis.

• OpenAI announcement of the benchmark and its expert-informed evaluation scope.

Medical-response leaderboard reporting overall and length-adjusted evaluation scores.

• Gemini 3.7 Flash result update on HealthBench Professional and MedAgentBench.

Agentic clinical-EHR benchmark in which models call simulated FHIR APIs to complete multi-step tasks across ten clinical task types.

• Grok 4.6 result update on the Medical Sphere leaderboard.
• Gemini 3.7 Flash result update, including multi-step EHR task performance.

Equity-research agent evaluation.

End-to-end enterprise document-work evaluation across life sciences, technology, due diligence, and legal analysis.

Internal evaluation of model writing quality, tone adherence, reasoning effort, and per-task cost.

Creative-work evaluation across real-world ideation, mockup, and refinement tasks, scored by working creatives; different models lead different phases.

Blind pairwise writing-style evaluation of 12 models across 108 texts and 1,647 judge decisions; lower AI-selection rates indicate less detectable slop.

Compares short stories written to constrained briefs with ten mandatory elements, using pairwise evaluator judgments for prose, originality, coherence, and characterization.

• September 2026 result update covering 50 models and 79,507 evaluator judgments.

Every's internal writing benchmark for comparing models in an editorial workflow; the linked update adds Gemini 3.8 Flash, Grok 4.6, and Muse Spark 1.3.

Internal editorial-voice evaluation reporting pairwise writing quality, model ranking, and cost per script.

Real-time voice-agent evaluation on grounded customer-service tasks.

• Launch post introduces realistic, controllable audio and deterministic scoring across customer-service tasks.

Six-provider, 240-case speech-to-text evaluation covering transcription quality, critical entities, downstream answer quality, and latency.

Live voice-agent benchmark reporting real-scenario pass rate, p99 latency, and model comparisons over time.

Benchmark for evaluating AI voice-cleaning systems that remove noise and improve spoken-audio clarity.

Healthcare-model comparison across HealthBench Hard, HealthBench Professional, AgentClinic, and MedAgentBench-v2.

Google's evaluation suite for MedGemma across clinical knowledge, radiology, dermatology, pathology, ophthalmology, and multimodal reasoning.

Open benchmark for long-horizon legal-agent work, evaluating whether agents can produce reliable legal work product across realistic workflows.

• Harvey's Tenet result update reports performance on LAB and related legal-agent benchmarks.

LAB extension with 500 contract-drafting, review, and negotiation tasks for in-house legal work.

Long-horizon corporate-law agent benchmark built from expert-authored work in simulated professional environments.

• Harvey cites APEX-Agents Corporate Law as an external legal-agent comparison.

Multi-turn contract-redlining benchmark that preserves native tracked changes and comments and grades legal correctness, negotiation quality, and deal-closing judgment.

• Harvey cites RedlineBench as a related agentic legal benchmark.

Open-ended, expert-authored benchmark for high-stakes professional reasoning in finance and law.

• Harvey cites Scale's Professional Reasoning Bench in its Tenet results.

Agentic context engineering: file operations, entity tracing, and long-horizon tool calls.

Leaderboard for finance-agent performance.

Writing and emotional-intelligence evaluation.

Arena-style evaluation for AI agents.

Document-understanding leaderboard across frontier models, open weights, and OCR solutions.

Long-horizon vending-machine agent evaluation, including deceptive-behavior analysis.

Spatial-reasoning benchmark: agents convert apartment photographs into accurate 2D floor plans.

Agents operating and reasoning over web user interfaces.

Computer-use and browser-use benchmark result for web-navigation agents.

Long-horizon browser computer-use benchmark with certified-optimal answers and verified compute.

Mobile-app agent benchmark covering 60 scenarios across two apps, with task completion and speed comparisons.

• August comparison of Haiku 4.5 and GPT-5.4-mini operating Argent and Agent Device on task completion, cost, and speed.

Browser-agent result comparison covering task performance, token use, and model cost.

Frontend coding leaderboard covering product, content-creation, simulation, gaming, and reference-based design tasks.

Verified customer-service agent tasks with tool use.

Finance-domain benchmark from Rogo.

Multi-agent Texas Hold'em evaluation that pits 17 frontier models against one another across more than 20,000 hands to test strategic reasoning under uncertainty.

• DM link to Vals' Poker Agent result report.

Open, long-horizon investment-workflow benchmark spanning research, events, and coverage monitoring.

• Gemini 3.6 Flash result with score, per-query cost, and runtime on the FrontierFinance harness.

FrontierFinance's GPT-5.6 Sol comparison across financial-analysis quality and cost per query.

Competitive mathematics evaluation and leaderboard.

Dynamic research-math benchmark asking models to recognize and refuse plausible but false statements, including recently refuted conjectures.

• Latest release shifts toward conjectures refuted in the previous month and runs models through their recommended harnesses.

Dynamic final-answer benchmark of research-level mathematics problems sourced from recent arXiv papers.

• Latest release focuses on recently refuted conjectures and runs models through harnesses with tools such as Python and SageMath.

Benchmarking agent skills and tool-use workflows.

Leaderboard for agent performance with MCP tools.

Research-agent retrieval evaluation reporting arXivQA recall and comparable-cost results.

Search-provider comparison using SealQA queries to evaluate retrieval quality and latency for agent workflows.

Compares native provider search with independent web-search tools on real-world tasks.

Two benchmarks for evaluating scientific-paper search and retrieval across a corpus of more than 300 million papers.

Web-research benchmark with 1,266 hard-to-locate factual questions, isolating the effect of search providers, tool configuration, and search budget.

FutureSearch pastcasting benchmark with 1,907 resolved binary and numeric forecasting questions researched against a frozen web corpus.

Web-research-agent benchmark with 169 diverse real-world tasks, offline search corpora, and curated reference answers.

FutureSearch's live bot evaluations across Metaculus tournaments, including rolling MiniBench contests and bot-versus-human comparisons.

Dynamic, contamination-resistant forecasting benchmark scored with a Brier Index on unresolved real-world questions.

Open benchmark for research agents that must search broadly and deeply.