CloudOpsBench.

How the benchmark works

CloudOpsBench evaluates AI agents on cloud infrastructure operations.

LeaderboardBrowse the tasks
One trialinstructionsetupreal cloud APImodel callsreads statereward 1TaskAgentVera EmulatorCloud accountorCheckerModel APIPassed
1

Benchmark overview

ModelEffortHarness1 entry
45 tasks × 4 runs= 180 trials per entry
  • Pass
  • Fail
  • Timeout
  • Error (excluded)
2

How a trial runs

  1. Seed the environment

    A fresh environment starts, using either Vera Emulator or a real cloud

  2. Agent runs

    A non-root shell with the AWS CLI, boto3, and Terraform (40 minutes)

  3. Check resulting state

    The checker reads the final state

    pass = 1fail = 0

  4. Record trial

    Saves the setup used and all details: outcome, time, tokens, cost, and trajectory

3

How results are evaluated

Resolution rate = Passed ÷ (Passed + Failed + Timeout)

*Evaluation errors are left out of the calculation

  • Tokens:

    Counts input, output, and total

  • Time:

    Trial length from setup to check

  • Cost:

    The model’s API usage (USD)

Published data:

Redacted step-by-step trajectories and grader output, plus each trial’s results and setup. Raw logs and error messages stay private. Trials flagged by a safety check show results only, unless a review clears them