Claude Outcomes: Rubric Grading for AI Agents

This blog is written by an agent. Not the ideas or the opinions, but the pipeline around them: research a keyword, draft, generate the hero image, build, publish, verify the URL returns 200. It runs on a cron. It has a QA checklist it's supposed to satisfy before it pushes.

One item on that checklist was “add the post to the blog index array.” The index reads its own array, separate from the post metadata file — an ugly bit of legacy in this codebase, but a real requirement. The agent skipped it. Then it skipped it again. It skipped it twenty times in a row, across 27 days, and every single run reported success, because the thing it was checking was “did the URL return 200?” and the URL always returned 200.

Twenty posts sat live, in the sitemap, reachable by direct link, and linked from nowhere on the site. I found it by accident and backfilled all twenty in one commit on July 31st.

Here's the part worth sitting with: the agent had the checklist. It read the checklist every run. It told itself it had done the work. An agent grading its own output isn't evaluating, it's remembering its own intent — and intent always looks like completion from the inside. That is precisely the hole Outcomes is shaped to fill.

An outcome tells a Managed Agents session what “done” looks like and how to measure it. You supply two things: a description of the task, and a rubric — a markdown document of per-criterion checks.

When the outcome is defined, the platform provisions a grader automatically. You don't write it, you don't deploy it, you don't pick its model. When the agent produces an artifact, the grader reads it against the rubric and returns an explanation of which criteria passed and which failed. That explanation goes back to the agent, which revises. The loop repeats until the rubric is satisfied or the iteration budget runs out.

The design decision that makes this more than a prompt trick: the grader gets a separate context window.It never sees the main agent's reasoning, its false starts, or the clever justification it constructed for why the missing section wasn't really necessary. It sees the rubric and the artifact. That's it.

Anthropic reports the feature lifted internal PowerPoint-generation quality by 10.1% and Word-document quality by 8.4% — with no model change and no prompt rewrite. Take vendor benchmarks as directional rather than gospel, but the direction is the interesting bit: that gain came purely from the architecture of who checks the work.

One event. You create a session against an existing agent and environment, then send user.define_outcome. The agent starts working the moment the event lands — there's no separate “now go” message.

client.beta.sessions.events.send(
    session_id=session.id,
    events=[
        {
            "type": "user.define_outcome",
            "description": "Build a DCF model for Costco in .xlsx",
            "rubric": {"type": "text", "content": RUBRIC},
            # or: {"type": "file", "file_id": rubric.id}
            "max_iterations": 5,   # optional; default 3, max 20
        }
    ],
)

Three fields do the work. description is the task. rubric is either inline text or a file_id from the Files API — upload it once and reuse it across sessions, which is what you want the moment the same rubric governs more than one run. max_iterations is optional, defaults to 3, and caps at 20.

Managed Agents endpoints need the managed-agents-2026-04-01 beta header. The official SDKs set it for you; raw cURL does not. You can also pass the define-outcome event in initial_events on session creation and start the whole thing in one call.

One outcome runs at a time. You can chain them by sending a new user.define_outcomeafter the previous one's terminal event, and the session keeps the earlier history.

Grading surfaces as three events: span.outcome_evaluation_start, an _ongoing heartbeat, and span.outcome_evaluation_end. The iteration field is zero-indexed — 0 is the first grade, 1 is the re-grade after the first revision.

The result on that end event is the one you branch on:

If you'd rather poll than stream, GET /v1/sessions/{session_id} exposes outcome_evaluations[].result, which reports pending, running or evaluating until an evaluation lands.

Two things to know going in. The grader's internal reasoning is opaque — you get the verdict and the explanation, not the deliberation. And the deliverables land in /mnt/session/outputs/ inside the sandbox; you fetch them through the Files API using the session ID as scope_id. They can take a few seconds to appear after the session goes idle, so a file that isn't listed yet isn't necessarily a file that failed to write.

This is where it lives or dies, and it's the part nobody wants to do properly.

The grader scores each criterion independently. That means a vague line doesn't produce a lenient grade, it produces a noisyone — passing on Tuesday and failing on Wednesday on the same artifact. “The analysis is thorough” is not a criterion. “The CSV contains a price column with numeric values” is.

My rule of thumb: if a stranger couldn't settle the question by looking only at the artifact, the grader can't either. Every criterion should name a thing you could point at.

If you're staring at a blank rubric, the shortcut in Anthropic's docs genuinely works better than writing from scratch: take an artifact you already know is good, hand it to Claude, ask what makes it good, then turn that analysis into criteria. You'll get specifics you would never have thought to write down.

And write the criterion you think is too obvious to state. The one my pipeline skipped for 27 days would have been a single line: the post's slug appears in the blog index array. Obvious. Unstated. Therefore ungraded.

The default is 3 and the ceiling is 20, and the instinct on reading that is to set it high because more retries sounds like better odds. It isn't. It's a budget, not a quality dial.

Every iteration is a full agent pass plus a full grader pass. Both bill tokens. On an outcome whose rubric contradicts the task and therefore can never pass, a cap of 10 doesn't buy you a better artifact — it buys you ten times the failure. And you won't notice on one session. You'll notice on the invoice, three weeks in, on the batch job you run nightly.

Budget 3 to 5 and treat a rising rate of max_iterations_reached as a defect report about your rubric. That result is the loop telling you the criteria are unreachable, mutually exclusive, or describing a different artifact than the one you asked for. Raising the cap silences the signal without fixing anything.

This is the same discipline that keeps any agent loop from eating a month of budget in a night — I wrote up the hard-stop pattern I use in cost controls for autopilot agents, and the same per-call arithmetic applies when you're paying for subagents. A grader is just another subagent with a very specific job.

The usage block on each span.outcome_evaluation_endevent breaks out input, output and cache tokens per evaluation. Log it. It's the only way to know what verification is actually costing you as a line item instead of a vibe.

Outcomes earns its cost when three things are true at once: the work produces an artifact, the quality bar is objective enough to enumerate, and a bad output is expensive to discover later. Financial models, generated reports, migration scripts, data extractions, client deliverables that go out with your name on them.

Skip it when:

• A deterministic check already exists. A compiler, a JSON schema validator, a test suite or an HTTP status code answers the question for a fraction of a cent and never has an opinion. Don't pay a language model to tell you whether the build passed.
• The output is conversational. Grading every turn of a chat is expensive and mostly meaningless.
• The criteria are genuinely subjective. “On brand” and “compelling” produce noise. If you can't write it as a check, a grader won't rescue it.

The honest framing: Outcomes is a paid, managed version of a pattern you can build yourself. The thing you're buying isn't the idea of a grader — it's the guarantee of context isolation, the automatic revision loop, and not having to run the orchestration. If you're already deep in a self-hosted stack, that may not be worth handing over. I broke down that hosted-versus-roll-your-own tradeoff more generally when comparing the Agent SDK against the raw Claude API, and the same logic holds here.

Outcomes is a Managed Agents feature. If you're on the Agent SDK, plain Messages API, or an orchestrator like n8n, you don't get the event stream — but you can copy the shape, and the shape is most of the value:

artifact = agent.run(task)

for i in range(MAX_ITERATIONS):          # 3, not 20
    # fresh client, fresh conversation, rubric + artifact ONLY.
    # do not pass the worker's messages in here.
    verdict = grader.evaluate(rubric, artifact)

    log(i, verdict.result, verdict.usage)
    if verdict.satisfied:
        break
    artifact = agent.revise(artifact, verdict.explanation)
else:
    alert("rubric never satisfied", rubric_id, artifact)

The one line people get wrong is the comment. If you reuse the worker's conversation for the grading call because it's convenient and the context is already warm, you've rebuilt self-grading and thrown away the entire benefit. Fresh context or don't bother.

And don't reach for a model at all when a script will do. In an n8n workflow a Code node checking three concrete conditions beats a grader call on cost, latency and determinism every time — I get into where that line falls in evaluating n8n AI agents. For code specifically, a deterministic hook that blocks a bad write is stricter than any rubric, because it can't be talked out of it.

1. Send one user.define_outcome event with a description, a rubric, and max_iterations.
2. Write criteria a stranger could verify from the artifact alone — including the one that's too obvious to state.
3. Set max_iterations to 3–5. Alert on max_iterations_reached instead of raising the cap.
4. Log the usage block on every evaluation so verification is a line item, not a vibe.
5. If you build it yourself, give the grader a fresh context window or you've built nothing.

The full field reference lives in Anthropic's Define outcomes documentation, and there's a runnable writer-plus-grader example in the Claude Cookbook.

My pipeline has a grader now. It checks the index array. The expensive lesson wasn't that my agent made a mistake — it's that it made the same mistake twenty times while reporting success, and nothing in the loop was structurally capable of noticing. A second pair of eyes that never saw your reasoning is worth more than a smarter model that saw all of it.