What cloud operations are, and what we test

2026-10-13

CloudOpsBench measures whether an AI agent can complete cloud operations work, as opposed to describing the steps. The tasks are meant to portray the day-to-day work of a DevOps engineer: the tickets that fill an ordinary week, not a set of puzzles. This post covers what that work is and what the tasks check.

What cloud operations work is

A cloud account is a large body of shared, long-lived state: storage, queues, networks, identities, access policies, alarms, schedules, deployed code. Operations is the work of changing that state safely while other things keep running on it. Very little of it starts from an empty account. It starts from what is already there, most of which belongs to someone else. For a DevOps engineer this is most of the job: cleaning up after a finished project, closing an access gap someone flagged, working out why something quietly stopped, making a small change before a maintenance window.

The requests are usually short and written in terms of intent. Here are three from the task set, lightly paraphrased:

• An application the company ran for its customers has been shut down. Take it out of this account completely. The billing workload also runs here and must still be in place when you are done.
• App servers in one private network time out connecting to the database in another. The link between the networks is active and the firewall rules were reviewed. Fix it.
• Urgent order events published to a notification topic are not reaching the on-call queue. Nothing is reporting an error. Find and fix every reason, without disturbing the other messages that queue already receives.

None of these says which resources are involved or which calls to make. The operator has to find out what exists, work out what the request means for this particular account, make the change, and leave everything else as it was.

What an agent is given

Each attempt starts in a fresh, isolated emulated cloud environment. A setup script builds the starting state before the agent runs: the resources the task is about and, usually, others that are not part of it. The tasks currently use AWS-style services.

The agent gets the task instruction, a shell, and the standard command-line and SDK tooling for the cloud API. A few tasks add a file to the workspace, such as an incident ticket or an existing infrastructure-as-code project that the change must go through. The agent is not given the setup script, the verifier, or the reference solution during the attempt, and has to discover the relevant resources through the API.

How an attempt is checked

The runner starts each attempt with its own emulator, lets the agent work, and then runs that task's verifier against the same emulator. The verifier asks the emulator what state the cloud resources are in and compares it with what the task required. It does not read the agent's final message. If every required check holds the attempt scores 1, otherwise 0. A setup failure or a verifier crash is recorded as an evaluation error, not as a failed attempt.

A verifier checks two things. The first is that what the task asked for actually changed, either by reading the configuration back or by exercising the system and watching what happens. The second is that everything outside the task is still as it was. An attempt that solves the problem but breaks or removes something unrelated does not pass.

The skills this tests

These are the habits that separate a careful engineer from a careless one on an ordinary day, and they are what a passing attempt needs.

• Discovery. Finding what a request refers to, including resources that a search by tag does not surface.
• Scope. Changing what the task covers and nothing else, in an account that also holds other teams' resources.
• Complete diagnosis. Several repair tasks have more than one cause, and fixing only one leaves the symptom unchanged.
• Working within constraints. Restoring access without granting a wildcard, or fixing delivery without recreating the resource.
• Finishing. Getting the environment into the required end state, which is the only thing the verifier looks at.
• And many more. This list is not exhaustive. Each task draws on whatever the work in front of it requires.

The tasks, including their setup scripts, verifiers and reference solutions, are public in the CloudOpsBench repository.