CloudOpsBench.

CloudOpsBench

A benchmark for AI agents on real cloud operations

How it worksView the tasks
Resolution rate versus Model cost (USD / trial)Each labeled square is a model and agent pair. Dark points are non-dominated across all models in the selected benchmark release. Light shading marks the area above each frontier. Error bars show 95% confidence intervals. Model links include exact values.0%20%40%60%80%100%$0$0.5$1$1.5$2$2.5$3Model cost (USD / trial)Resolution Ratekimi-k3.1-pro (high) / terminus-2: 19.1%, $0.0597 per trialkimi-k3.1-pro (high)terminus-219.1% · $0.0597grok-4.6 (medium) / grok-build: 36.7%, $0.2846 per trialgrok-4.6 (medium)grok-build36.7% · $0.2846grok-4.6 (high) / grok-build: 36.1%, $0.3426 per trialgrok-4.6 (high)grok-build36.1% · $0.3426qwen3.8-max (high) / terminus-2: 33.7%, $0.3435 per trialqwen3.8-max (high)terminus-233.7% · $0.3435grok-4.6 (high) / cursor-cli: 39.7%, $0.3544 per trialgrok-4.6 (high)cursor-cli39.7% · $0.3544claude-opus-5-5 (high) / claude-code: 46.7%, $0.2031 per trial — Pareto frontierclaude-opus-5-5 (high)claude-code46.7% · $0.2031gemini-3.8-flash (high) / terminus-2: 42.8%, $0.3891 per trialgemini-3.8-flash (high)terminus-242.8% · $0.3891grok-4.7 (medium) / grok-build: 40.0%, $0.5089 per trialgrok-4.7 (medium)grok-build40.0% · $0.5089gpt-6-astra (high) / terminus-2: 42.8%, $0.5255 per trialgpt-6-astra (high)terminus-242.8% · $0.5255grok-4.7 (high) / grok-build: 48.3%, $0.7070 per trialgrok-4.7 (high)grok-build48.3% · $0.7070claude-opus-5-5 (high) / terminus-2: 47.8%, $0.2504 per trial — Pareto frontierclaude-opus-5-5 (high)terminus-247.8% · $0.2504gemini-4-argon (high) / gemini-cli: 57.5%, $0.5584 per trial — Pareto frontiergemini-4-argon (high)gemini-cli57.5% · $0.5584grok-4.7 (high) / cursor-cli: 56.5%, $0.7713 per trialgrok-4.7 (high)cursor-cli56.5% · $0.7713gpt-6-astra (high) / codex: 54.2%, $1.0467 per trialgpt-6-astra (high)codex54.2% · $1.0467claude-fable-5-1 (high) / terminus-2: 43.8%, $1.7729 per trialclaude-fable-5-1 (high)terminus-243.8% · $1.7729deepseek-v4.1-flash (high) / terminus-2: 21.8%, $0.0414 per trial — Pareto frontierdeepseek-v4.1-flash (high)terminus-221.8% · $0.0414gpt-6-astra (max) / codex: 58.9%, $1.7271 per trial — Pareto frontiergpt-6-astra (max)codex58.9% · $1.7271claude-fable-5-1 (max) / claude-code: 60.6%, $2.2460 per trial — Pareto frontierclaude-fable-5-1 (max)claude-code60.6% · $2.2460
How to read this chart

Usage is averaged per trial, including errors. Error bars are 95% confidence intervals. Hover over or tab to a model to isolate it and see its exact values.

One frontier compares all models in the selected benchmark release, regardless of harness, reasoning effort, task coverage, or evaluation pins. A model is on the frontier when no other model has lower usage with equal or higher resolution, or higher resolution with equal usage.