Darkmind / Issue 026

Give Every Agent Loop a Stop Contract

Define the bounded outcomes, budgets, and evidence an agent loop must return before it starts, so “keep going” never becomes an accidental policy.

Nov 2, 20264 min readAgent reliability

An agent loop often begins with a harmless rule: if the model asks for a tool, run it and continue. Without an explicit stopping policy, “continue” can quietly become “keep searching until context fills” or “take one more tool call after the evidence is enough.” The costly failure is an agent with no defined point at which it must return a bounded, honest result.

The practical move is to give every agent run a stop contract: allowed terminal outcomes, budgets, evidence for a success claim, and the receipt to return when it stops short. The runtime—not the model—enforces it.

OpenAI describes an agent run as a loop that continues when the model produces tool calls or handoffs and returns when it produces a final answer with no more tool work.[1] Its guidance also calls out max-turn limits, guardrail exceptions, and tool errors as runtime outcomes that need deliberate handling.[1] Anthropic makes the same operational distinction through stop_reason: a response might end naturally, request a tool, reach a token limit, pause after a server-tool iteration limit, or exceed the context window.[2] In other words, a stopped run is not necessarily a completed task. Your application needs a policy for what each ending means.

Separate “the model stopped” from “the work is complete”

A natural-language final answer is one signal, not a completion check. For a research agent, success may require two eligible sources and a cited answer. For a reconciliation agent, it may require scanning a named input set, reporting failures separately, and returning stable IDs for every match. For a publishing agent, success may require a build result and a live verification receipt.

Write that evidence into the stop contract. Then define non-success endings as first-class results: needs_input, needs_approval, budget_exhausted, blocked, partial, or failed. These are more useful than either a silent loop or a made-up “done.” A model may suggest what to try next, but it should not be able to extend its own time, tool-call, cost, scope, or effect budget.

Copy this stop contract

Place this in the application layer around one agent run. Pick values that fit the job and risk; a ten-call research lookup and a two-call production update should not inherit the same limit.

RUN INTENT
- run_id: [server-issued ID]
- requested outcome: [one observable result]
- authorized scope: [corpus, tenant, repository, or target IDs]
- allowed tools: [small named allowlist]
- prohibited effects: [send | spend | publish | delete | permission change]

SUCCESS IS TRUE ONLY WHEN
- required evidence: [e.g., two source IDs / test receipt / provider result ID]
- required validation: [schema check / policy gate / human approval]
- final claim or action matches the authorized scope

BUDGETS ENFORCED OUTSIDE THE MODEL
- max_model_turns: [N]
- max_tool_calls: [N], by tool and total
- max_elapsed_seconds: [N]
- max_cost_or_tokens: [N]
- max_pages_or_records: [N]
- max_external effects: [0 or named limit]

TERMINAL OUTCOMES
- completed: success conditions and evidence are present
- partial: useful bounded result; completion condition is not met
- needs_input: a required scope, ID, or decision is missing
- needs_approval: an allowed next action requires confirmation
- budget_exhausted: a named limit was reached
- blocked: access, policy, dependency, or tool failure prevents progress
- failed: an unrecoverable runtime or validation error occurred

RETURN RECEIPT
{
  "run_id": "...",
  "outcome": "completed | partial | needs_input | needs_approval | budget_exhausted | blocked | failed",
  "completion_checks": [{"name":"...","passed":true|false,"evidence_ref":"..."}],
  "usage": {"model_turns":0,"tool_calls":0,"elapsed_seconds":0,"records_scanned":0},
  "scope_used": "...",
  "results": ["stable IDs, source references, or artifact paths"],
  "unmet_condition": "... or null",
  "next_safe_action": "stop | ask_for_input | request_approval | resume_with_new_budget"
}

The key is that partial must be honest. If a research run inspected one page of a paginated collection, its receipt can say what it found and that exhaustive coverage was not achieved. It must not claim “no other records exist.” If a tool times out after a write, the receipt should preserve the operation reference and say that the result is unknown, not label it failed and retry with a new identity.

Make continuation an explicit new decision

A stopped run may be resumable, but resuming should not mean blindly replaying the original loop. OpenAI recommends resuming an approval pause from the same state rather than starting a fresh turn, which preserves the run’s continuity.[1] Anthropic similarly documents continuing a paused server-tool turn with the existing conversation state, while its examples bound the number of continuation requests.[2]

Apply the same discipline to your own limits. A resume needs a reason: a human supplied the missing ID, approved the proposal, widened the authorized corpus, or granted a new budget. Store that reason with the original receipt. Keep the earlier counters and scope visible; otherwise an agent can evade a ten-call cap by treating every ten calls as a new run.

There is one important exception: a technical continuation is not automatically an authorization continuation. If the task’s target, payload, or effect changes, start a new proposal and run rather than reusing a prior budget or approval.

Failure mode: limits that hide useful work

A very tight cap can make agents look flaky and encourage developers to automatically rerun them. A very loose cap turns a limit into theater. Start from the smallest budget that can finish a representative, safe task; measure where runs stop; then adjust per workflow.

The habit to keep

do not ask an agent merely to finish. Define what counts as finished, the resources it may spend getting there, and the precise receipt it must return when it cannot.

Sources

  1. OpenAI, Running agents
  2. Anthropic, Stop reasons and fallback