CloudOpsBench.
THE CLOUD AGENT BENCHMARK v0.2

Real cloud tasks.
Proven capability.

An evaluation of AI agents on cloud operations. Reproducible tasks, isolated infrastructure, and verifiable outcomes.

Explore results ↓How the benchmark works ↗
POWERED BY VERA EMULATOR

The cloud environment behind CloudOpsBench. Agents work with emulated AWS resources in isolated, inspectable environments.

Explore Vera Emulator ↗
THE EVALUATION CYCLEFIG. 01
Same conditions. Observable outcomes.
THE RESULTS

Leaderboard

● Production
2 model / agent pairsv0.2AWS / emulatedSelect any row to explore
Pass · 6Fail · 6Timeout · 0Error · 12
Tasks by model and harness. Each colored square links to one recorded trial.
Taskclaude-opus-5-5claude-codegpt-6-astracodex
aws/promote-lambda-live-alias
Run 1
claude-opus-5-5 / claude-code · aws/promote-lambda-live-alias · Trial 2 · Evaluation error · Run 20260928T201252Z-615ea26a490c4dd58ddbe0cbb6c23549claude-opus-5-5 / claude-code · aws/promote-lambda-live-alias · Trial 1 · Evaluation error · Run 20260928T201252Z-615ea26a490c4dd58ddbe0cbb6c23549
Run 2
claude-opus-5-5 / claude-code · aws/promote-lambda-live-alias · Trial 2 · Evaluation error · Run 20260928T214043Z-bd38bd0b97d943aa8074f649590cbf6dclaude-opus-5-5 / claude-code · aws/promote-lambda-live-alias · Trial 1 · Evaluation error · Run 20260928T214043Z-bd38bd0b97d943aa8074f649590cbf6d
Run 3
claude-opus-5-5 / claude-code · aws/promote-lambda-live-alias · Trial 1 · Passed · Run 20260928T223718Z-2ccb683a4461403aa9a0c21ce2ea0dd0claude-opus-5-5 / claude-code · aws/promote-lambda-live-alias · Trial 2 · Passed · Run 20260928T223718Z-2ccb683a4461403aa9a0c21ce2ea0dd0
gpt-6-astra / codex · aws/promote-lambda-live-alias · Trial 2 · Passed · Run 20260928T225843Z-bea4fe9ae71b45498939ac2167edd512gpt-6-astra / codex · aws/promote-lambda-live-alias · Trial 1 · Passed · Run 20260928T225843Z-bea4fe9ae71b45498939ac2167edd512
aws/tighten-kinesis-resource-policy
Run 1
claude-opus-5-5 / claude-code · aws/tighten-kinesis-resource-policy · Trial 2 · Evaluation error · Run 20260928T201252Z-615ea26a490c4dd58ddbe0cbb6c23549claude-opus-5-5 / claude-code · aws/tighten-kinesis-resource-policy · Trial 1 · Evaluation error · Run 20260928T201252Z-615ea26a490c4dd58ddbe0cbb6c23549
Run 2
claude-opus-5-5 / claude-code · aws/tighten-kinesis-resource-policy · Trial 2 · Evaluation error · Run 20260928T214043Z-bd38bd0b97d943aa8074f649590cbf6dclaude-opus-5-5 / claude-code · aws/tighten-kinesis-resource-policy · Trial 1 · Evaluation error · Run 20260928T214043Z-bd38bd0b97d943aa8074f649590cbf6d
Run 3
claude-opus-5-5 / claude-code · aws/tighten-kinesis-resource-policy · Trial 2 · Passed · Run 20260928T223718Z-2ccb683a4461403aa9a0c21ce2ea0dd0claude-opus-5-5 / claude-code · aws/tighten-kinesis-resource-policy · Trial 1 · Passed · Run 20260928T223718Z-2ccb683a4461403aa9a0c21ce2ea0dd0
gpt-6-astra / codex · aws/tighten-kinesis-resource-policy · Trial 1 · Failed · Run 20260928T225843Z-bea4fe9ae71b45498939ac2167edd512gpt-6-astra / codex · aws/tighten-kinesis-resource-policy · Trial 2 · Failed · Run 20260928T225843Z-bea4fe9ae71b45498939ac2167edd512
aws/tighten-sqs-redrive-allow-policy
Run 1
claude-opus-5-5 / claude-code · aws/tighten-sqs-redrive-allow-policy · Trial 1 · Evaluation error · Run 20260928T201252Z-615ea26a490c4dd58ddbe0cbb6c23549claude-opus-5-5 / claude-code · aws/tighten-sqs-redrive-allow-policy · Trial 2 · Evaluation error · Run 20260928T201252Z-615ea26a490c4dd58ddbe0cbb6c23549
Run 2
claude-opus-5-5 / claude-code · aws/tighten-sqs-redrive-allow-policy · Trial 2 · Evaluation error · Run 20260928T214043Z-bd38bd0b97d943aa8074f649590cbf6dclaude-opus-5-5 / claude-code · aws/tighten-sqs-redrive-allow-policy · Trial 1 · Evaluation error · Run 20260928T214043Z-bd38bd0b97d943aa8074f649590cbf6d
Run 3
claude-opus-5-5 / claude-code · aws/tighten-sqs-redrive-allow-policy · Trial 2 · Failed · Run 20260928T223718Z-2ccb683a4461403aa9a0c21ce2ea0dd0claude-opus-5-5 / claude-code · aws/tighten-sqs-redrive-allow-policy · Trial 1 · Failed · Run 20260928T223718Z-2ccb683a4461403aa9a0c21ce2ea0dd0
gpt-6-astra / codex · aws/tighten-sqs-redrive-allow-policy · Trial 1 · Failed · Run 20260928T225843Z-bea4fe9ae71b45498939ac2167edd512gpt-6-astra / codex · aws/tighten-sqs-redrive-allow-policy · Trial 2 · Failed · Run 20260928T225843Z-bea4fe9ae71b45498939ac2167edd512

One square per trial. Click for details; hover for usage. — means no recorded trials. Multiple runs stay separate.

How outcomes are classified

This is trial history, not just the latest summary. Orange means the agent actually hit its execution timeout and did not pass; setup, verifier, infrastructure and emulator failures are red. A long duration alone is not a timeout. Gray means the grader returned a failing result without an execution error.

Resolution rate = passed / valid attempts. Evaluation errors excluded.2 entries / v0.2
What’s behind a score?How the benchmark works ↗