Darkmind / Issue 043

Pair the Near Miss With the Pass

Turn a production mistake into a paired contrast case that changes one decision-relevant fact, so an AI evaluation can show whether a fix learned the boundary or merely memorized the example.

Nov 19, 20264 min readAI evaluation

A production failure is a valuable test case—but a single bad answer often cannot tell you what broke. Did the model miss a required fact? Did your prompt leave the rule vague? Did a new tool description redirect it? Or did the evaluator simply dislike the wording?

When one incident becomes one regression fixture, it is easy to celebrate the patch that makes that exact input pass. The harder question is whether the system now recognizes the decision boundary that the incident exposed.

Use a paired contrast case to answer it. Keep a known failing input, then make one companion input that changes exactly one decision-relevant fact and should receive a different result. Evaluate the pair together. If the model gets both right, the change has evidence of respecting the boundary. If it fixes one and fails the other, you have a precise next debugging target—not a larger pile of examples.

OpenAI recommends task-specific evaluations that reflect real-world distributions, including typical, edge, and adversarial cases, and notes that comparison, classification, and scoring against defined criteria are more reliable evaluation shapes than open-ended generation.[2] Anthropic likewise recommends task-specific cases, explicit success criteria, and held-out evaluations before shipping.[3]

Turn one incident into a contrast pair

Start with the actual failing request and write the expected observable behavior. Do not begin by asking a model to invent “similar tests.” First identify the smallest fact that should have changed the outcome.

For a support-routing agent, the failure might be:

“My card was charged twice for order 8472” was sent to the general FAQ response instead of the billing-dispute flow.

The paired case is not a random paraphrase. It preserves the customer tone, the order reference, and the request for help, but changes the causal fact:

“My card was declined twice for order 8472” should go to the payment-troubleshooting flow, not billing dispute.

The pair asks a crisp question: does the route depend on duplicate charge rather than superficial overlap such as “card,” “twice,” or the order number?

Keep the edit minimal. Change one of these at a time:

This is not a request to make every test synthetic. The original incident remains valuable because it carries real wording and context. Its partner makes the intended rule falsifiable.

Copy this incident-to-pair card

Use this card when adding a regression test after an agent, retrieval, or output failure.

INCIDENT-TO-PAIR CARD

case_id: [stable identifier]
production_case:
  input: [sanitized failing input]
  expected_outcome: [label, tool choice, required field, or refusal]
  observed_outcome: [what actually happened]

boundary:
  rule: [the decision rule this case is supposed to exercise]
  decisive_fact: [one fact that should control the outcome]

contrast_case:
  input: [same context; change only decisive_fact]
  expected_outcome: [different, explicit outcome]
  changed_field: [exact span or structured field]
  unchanged_context: [what must remain equivalent]

grading:
  production_check: [exact match / schema assertion / rubric]
  contrast_check: [same grading method]
  pair_rule: pass only when both expected outcomes hold

release_note:
  owner: [person or team]
  prompt_or_workflow_version: [version]
  failure_action: [block / investigate / accept with reason]

Prefer the cheapest reliable grader. A route, tool choice, status field, or required citation can usually be checked in code. Use a human or model-based rubric only for the part that truly needs judgment, and record that rubric alongside the pair. Anthropic recommends choosing the fastest reliable scalable grading method and making LLM-grader outputs empirical or specific.[3]

Make failures diagnostic, not merely red

Run both cases against the baseline and the proposed change. Then label the result:

Add the pair to a held-out suite after you finish debugging. Do not repeatedly tune against it until it passes and then describe it as independent evidence. OpenAI’s eval guidance separates collecting a representative dataset, defining metrics, comparing results, and continuously growing the set from observed production behavior.[1][2]

Failure mode: changing three things at once

A “nearby” case that changes the user role, date, document status, and wording may be useful for broad coverage, but it cannot diagnose the boundary. If the outputs differ, you will not know which change mattered. Split it into several pairs, or call it a scenario test rather than a contrast test.

Also do not force a binary pair onto an inherently graded task such as writing quality. In that case, pair a concrete rubric criterion—such as whether the response preserves an uncertainty statement—with a narrow edit that should flip that criterion. The technique is strongest where a workflow has a consequential decision to make.

The habit to keep

after every meaningful AI failure, preserve the real case and create one minimal companion that should behave differently. A repair that survives both has learned more than a memorized answer.

Sources

  1. OpenAI, “Working with evals”
  2. OpenAI, “Evaluation best practices”
  3. Anthropic, “Define success criteria and build evaluations”