Benchmark catalogue

Cohere North Micro Vision result report covering document understanding, visual question answering, and other visual-understanding benchmarks.

• Cohere's launch post summarizes the model's visual benchmark positioning.

Vision-language model result post reporting ScreenSpot-v2, RealWorldQA, TextVQA, RefCOCO, and ToolSandbox scores.

• LFM2.5-VL-3B model card and release materials.

Model-configuration guidance comparing coding performance and cost across reasoning-effort levels.

Empirical evaluation reporting that test-driven development harmed coding-agent performance.

Model comparison on self-contained HTML5 physics demos, including token use, cost, and simulation quality.

Harness and model comparison on Databricks' multi-million-line internal codebase.

References ProgramBench, CodeClash, AlgoTune, SWEfficiency, CritPt, and VideoGameBench.

Results across Vals Index, ProgramBench, ProofBench, VibeCodeBench, and Public Benefits Bench.

Comparison of three current model leaderboards.

Model results broken down by production-task category.

Gemini 3 Pro results across Vals benchmarks.

Discussion of tougher coding benchmarks beyond SWE-bench.

Internal evaluation of document and image data-extraction accuracy across 40,000 fields.

Real-world agentic coding comparison across codebases.

High-level software-engineering requests with hidden tests and validation agents.

Production-code quality benchmark measuring whether a maintainer would merge an agent-generated pull request.

Resolving real GitHub issues in Python repositories.

Warning OpenAI reports that SWE-bench Verified is increasingly contaminated and recommends SWE-bench Pro instead.

Scale AI's public leaderboard for a harder SWE-bench variant.

A reproducible software-engineering benchmark.

Continuously refreshed real-world software-engineering tasks.

Efficiency-focused evaluation for software-engineering agents.

• Launch post: 498 optimization tasks across nine data-science, ML, and HPC repositories.

Long-horizon software-engineering benchmark.

Long-horizon whole-repository migration benchmark covering language, framework, platform, and build-toolchain rewrites, graded for migration completeness, behavior, and hidden regressions.

• Launch post reports 20 migrations, 520 runs, and 28 accepted runs (5.4%).
• Official benchmark homepage and leaderboard.
• Paper describing the three-stage audit, behavioral, and agentic-verification protocol.

Interactive coding sessions scored for correctness and user correction.

Software-engineering benchmark with cost and performance comparisons between frontier models.

Private-codebase coding-agent evaluation comparing quality, cost, and speed across models and harnesses.

A demanding software-engineering benchmark.

Production software-engineering tasks spanning deployment, integration, and debugging.

Ultra-long-horizon software-engineering benchmark with 20-hour runs; v2 expands the suite to 34 difficult engineering and research tasks.

• FrontierSWE v2 release notes and updated leaderboard.
• Proximal's announcement of the expanded task suite and methodology.
• Independent result post reporting model performance on v2.

Program-level coding tasks and evaluation.

• Opus 5 model-card experiment comparing async and flat multi-agent harnesses with a single-agent baseline.
• Gemini 3.5 and 3.6 Flash benchmark results for binary reimplementation without source, decompilation, or internet access.

Long-horizon coding benchmark where agents reimplement complete programs from behavior without their source code.

Iterative coding benchmark that measures correctness, code erosion, and verbosity as agents extend their own solutions across changing requirements.

• Earlier result post introducing the benchmark.
• September 2026 update: GPT-6 Astra entered the benchmark.

Repository-scale coding benchmark.

Contamination-resistant evaluation of code generation.

Result analysis of coding-agent performance across novel interactive environments and programming languages.

Custom coding evaluation that turns merged pull requests into tasks and measures an agent's resolution rate.

Coding-agent leaderboard with model result updates.

• Result update reporting GPT-5.3-Codex at 86% on IBench.

Coding-agent result comparison covering task performance, cost efficiency, code density, and review quality.

• Grok 4.6 result update on GBENCH's complex multi-agent coding evaluations.

One-shot coding-agent ranking and model comparison.

Agentic coding benchmark and leaderboard.

Scientific-reproducibility tasks that require agents to run and interpret research repositories.

Coding-agent evaluation on realistic React work, including production quality, performance, and accessibility criteria.

Google's Android-specific coding benchmark and leaderboard covering 100 real-world Android development tasks, with quality, latency, and cost reporting.

• Android Developers' update adds new frontier models to the leaderboard.

Game-development agent benchmark result comparing model task-success rates.

Vibe-coding benchmark covering debugging, refactoring, UI, security, cost, and speed.

End-to-end web-app benchmark that assesses whether agents meet product requirements through behavioral test plans.

Next.js's evaluation suite for AI coding agents.

Terminal-agent benchmark and leaderboard for real command-line tasks.

Community benchmark for frontier agent work, built by the team behind Terminal-Bench and Harbor.

Compares coding-agent harnesses by speed, cost per solve, and task performance.

Internal coding-agent evaluation on Databricks' multi-million-line codebase, comparing quality and cost.

Independent real-task evaluations caution that context-compression tools can increase token cost when they force rereads or break cache reuse.

• Copilot evaluation: neutral-to-higher token use after compression.
• Hermes evaluation: most Headroom changes increased token cost, with one file-search exception.
• Claude Code real-task benchmark: 17% higher token cost from cache disruption.
• RTK evaluation and analysis of command-output token reduction for coding agents.

Compares sandbox providers on repeatable developer and CI/CD workloads, including end-to-end workflow time.

Enterprise-agent benchmark that measures task performance under realistic EDR, SASE, and DLP policy constraints.

Code-review agent evaluation based on curated, substantive findings from real merged pull requests.

Open-source comparison of code-review agents, measuring bug detection, code improvements, and developer adoption.

Evaluation of AI code-review performance.

Code-review comparison on 400 real pull requests.

Security-agent benchmark spanning 869 real vulnerabilities across userspace software, V8, and the Linux kernel.

Fresh-CVE security-agent benchmark testing vulnerability rediscovery, recall, consistency, and cost across 32 recent vulnerabilities and three runs per model.

• Aikido's launch post summarizes the 11.7-billion-token evaluation and model results.
• Earlier known-CVE benchmark using 26 vulnerabilities and pass@3 pooling.

Security-agent benchmark comparison covering accuracy, price-normalized precision, and model cost.

Contamination-resistant software reverse-engineering benchmark with 262 binary instances and 1,572 deterministically graded tasks.

• Vals AI's announcement frames SRE-Bench as a test of understanding binaries without source code.

Agentic GPU-kernel benchmark and leaderboard for writing correct, efficient CUDA kernels.

Cross-provider model comparisons, including coding and speed.

Registry and capability tracker for AI benchmarks, models, and their release history.

Ground-truth vision benchmark across object detection, counting, identification, OCR, data extraction, and image reasoning, with cost and speed reporting.

• Roboflow's earlier GPT-5.6 VLM benchmark report.
• Cost-sorted model comparison from the current Vision Evals leaderboard.
• September 2026 refresh adds Gemini 3.8 Flash and Fable 5.1 to the top 10; Roboflow notes Fable is about 50% faster than the Gemini models ahead of it.
• Vision-model detection deep dive highlighting detail, box precision, OCR, speed, and cost.

Controlled lexical-fingerprint experiment comparing models' use of signature words across identical questions; the linked ox-alpha page analyzes 1,469 complete responses against 374 comparison models.

Frequently refreshed benchmark designed to reduce contamination.

Collection of AI benchmark analyses and datasets.

Cross-benchmark model leaderboard combining quality, cost, speed, context, and confidence data.

Scale AI's general model-evaluation leaderboard.

Kaggle's benchmark hub and leaderboards.

Live, community-voted LLM leaderboard with real-time Vibe Score and Curse Trend metrics.

Practitioner-panel benchmark that compares model releases through engineers' real-world, substantive use.

Aggregate evaluation of model performance across tasks.

Measures whether agents learn from feedback during repeated play.

Continuously rotated, sealed evaluations designed to reduce benchmark gaming and training contamination.

Difficult multimodal reasoning benchmark covering spatial reasoning, visual understanding, and multi-step reasoning across images.

Multimodal spatial-reasoning benchmark covering complex 2D and 3D tasks, with a public model leaderboard and human baseline.

• Original benchmark announcement and SpatialBench description.
• September 2026 update says GPT-6 Astra saturated the spatial-reasoning evaluation.

Post-training-agent benchmark with anti-cheat checks for data contamination, model substitution, API use, and trace lookup.

Adversarial multi-turn debate evaluation with side-swapped matchups and multi-model judging of factual accuracy, rebuttals, and sustained coherence.

• September 2026 result update describing the current Debate Benchmark leaderboard.

Game-playing evaluation comparing a vision-language model with professional GeoGuessr players.

Long-horizon agent learning across scientific, professional, coding, math, and game tasks.

General model-capability benchmark with a public high-score tracker.

Knowledge and hallucination evaluation from Artificial Analysis.

Measures whether models detect and clearly challenge flawed or misleading premises across domains and techniques.

Reasoning and visual abstraction benchmark comparison.

Comparison of top ARC-AGI-2 performance and cost efficiency for GPT-5.6 Sol.

Game-like evaluation of how well models orient in unfamiliar situations; includes verified frontier-model results.

OpenAI long-context reasoning benchmark for breadth-first graph traversal over very large inputs.

Visual-reasoning benchmark result tracking a new high score and performance by reasoning level.

WeirdML comparison of model performance, result consistency, and cost at a near-saturated frontier score.

Framer's 236-task evaluation of AI agents building and maintaining responsive websites, spanning layout, interactions, and design workflows.

Framer's continuous evaluation of translation models for quality, structural correctness, latency, and cost on real website content.

Multi-step business-automation workflows with cost per task.

Internal production-harness evaluation of agent performance on everyday and long-horizon spreadsheet tasks.

Sales-call coaching benchmark scored against hidden ground truth across synthetic calls.

Ongoing production evaluation of models on anonymized Notion knowledge-work traffic, scored for resolution rate, cost per task, and time; Notion explicitly describes it as a live eval rather than a traditional fixed benchmark.

• Launch post describing the live anonymized-traffic evaluation and ensemble judge.

Hex's frontier benchmark for agentic analytics: 100 realistic Q&A and open-ended data-work tasks in a synthetic warehouse, scored for analytical judgment rather than code generation alone.

• Announcement of DataBench as a benchmark derived from realistic Hex usage rather than pub-trivia questions.

Medical-response leaderboard reporting overall and length-adjusted evaluation scores.

• Gemini 3.7 Flash result update on HealthBench Professional and MedAgentBench.

Agentic clinical-EHR benchmark in which models call simulated FHIR APIs to complete multi-step tasks across ten clinical task types.

• Grok 4.6 result update on the Medical Sphere leaderboard.
• Gemini 3.7 Flash result update, including multi-step EHR task performance.

Equity-research agent evaluation.

End-to-end enterprise document-work evaluation across life sciences, technology, due diligence, and legal analysis.

Internal evaluation of model writing quality, tone adherence, reasoning effort, and per-task cost.

Blind pairwise writing-style evaluation of 12 models across 108 texts and 1,647 judge decisions; lower AI-selection rates indicate less detectable slop.

Compares short stories written to constrained briefs with ten mandatory elements, using pairwise evaluator judgments for prose, originality, coherence, and characterization.

• September 2026 result update covering 50 models and 79,507 evaluator judgments.

Every's internal writing benchmark for comparing models in an editorial workflow; the linked update adds Gemini 3.8 Flash, Grok 4.6, and Muse Spark 1.3.

Internal editorial-voice evaluation reporting pairwise writing quality, model ranking, and cost per script.

Real-time voice-agent evaluation on grounded customer-service tasks.

Six-provider, 240-case speech-to-text evaluation covering transcription quality, critical entities, downstream answer quality, and latency.

Live voice-agent benchmark reporting real-scenario pass rate, p99 latency, and model comparisons over time.

Benchmark for evaluating AI voice-cleaning systems that remove noise and improve spoken-audio clarity.

Healthcare-model comparison across HealthBench Hard, HealthBench Professional, AgentClinic, and MedAgentBench-v2.

Google's evaluation suite for MedGemma across clinical knowledge, radiology, dermatology, pathology, ophthalmology, and multimodal reasoning.

Open benchmark for long-horizon legal-agent work, evaluating whether agents can produce reliable legal work product across realistic workflows.

• Harvey's Tenet result update reports performance on LAB and related legal-agent benchmarks.

LAB extension with 500 contract-drafting, review, and negotiation tasks for in-house legal work.

Long-horizon corporate-law agent benchmark built from expert-authored work in simulated professional environments.

• Harvey cites APEX-Agents Corporate Law as an external legal-agent comparison.

Multi-turn contract-redlining benchmark that preserves native tracked changes and comments and grades legal correctness, negotiation quality, and deal-closing judgment.

• Harvey cites RedlineBench as a related agentic legal benchmark.

Open-ended, expert-authored benchmark for high-stakes professional reasoning in finance and law.

• Harvey cites Scale's Professional Reasoning Bench in its Tenet results.

Agentic context engineering: file operations, entity tracing, and long-horizon tool calls.

Leaderboard for finance-agent performance.

Writing and emotional-intelligence evaluation.

Arena-style evaluation for AI agents.

Document-understanding leaderboard across frontier models, open weights, and OCR solutions.

Long-horizon vending-machine agent evaluation, including deceptive-behavior analysis.

Spatial-reasoning benchmark: agents convert apartment photographs into accurate 2D floor plans.

Agents operating and reasoning over web user interfaces.

Computer-use and browser-use benchmark result for web-navigation agents.

Long-horizon browser computer-use benchmark with certified-optimal answers and verified compute.

Mobile-app agent benchmark covering 60 scenarios across two apps, with task completion and speed comparisons.

Browser-agent result comparison covering task performance, token use, and model cost.

Frontend coding leaderboard covering product, content-creation, simulation, gaming, and reference-based design tasks.

Verified customer-service agent tasks with tool use.

Finance-domain benchmark from Rogo.

Multi-agent Texas Hold'em evaluation that pits 17 frontier models against one another across more than 20,000 hands to test strategic reasoning under uncertainty.

• DM link to Vals' Poker Agent result report.

Open, long-horizon investment-workflow benchmark spanning research, events, and coverage monitoring.

• Gemini 3.6 Flash result with score, per-query cost, and runtime on the FrontierFinance harness.

FrontierFinance's GPT-5.6 Sol comparison across financial-analysis quality and cost per query.

Competitive mathematics evaluation and leaderboard.

Benchmarking agent skills and tool-use workflows.

Leaderboard for agent performance with MCP tools.

Research-agent retrieval evaluation reporting arXivQA recall and comparable-cost results.

Search-provider comparison using SealQA queries to evaluate retrieval quality and latency for agent workflows.

Compares native provider search with independent web-search tools on real-world tasks.

Two benchmarks for evaluating scientific-paper search and retrieval across a corpus of more than 300 million papers.

Web-research benchmark with 1,266 hard-to-locate factual questions, isolating the effect of search providers, tool configuration, and search budget.

FutureSearch pastcasting benchmark with 1,907 resolved binary and numeric forecasting questions researched against a frozen web corpus.

Original FutureSearch pastcasting benchmark with hard forecasting questions, frozen web corpora, and strategic-reasoning evaluation.

Web-research-agent benchmark with 169 diverse real-world tasks, offline search corpora, and curated reference answers.

FutureSearch's live bot evaluations across Metaculus tournaments, including rolling MiniBench contests and bot-versus-human comparisons.

Dynamic, contamination-resistant forecasting benchmark scored with a Brier Index on unresolved real-world questions.

Open benchmark for research agents that must search broadly and deeply.