Darkmind / Issue 027

Canary a Model Change Before You Switch

Treat a new model ID, alias update, or provider behavior shift as a release: pin the baseline, shadow a bounded sample, compare outcome checks, and promote only through a stop rule.

Nov 3, 20265 min readModel operations

A model upgrade can look like a harmless configuration edit: replace one ID, accept a provider alias update, or enable a newly available capability. But a model is part of the behavior of a working system. The same prompt can yield a different classification, tool argument, refusal, citation choice, or amount of extra prose. If that output drives a customer queue, a retrieval answer, or an action proposal, changing the model without a release gate turns ordinary traffic into the test set.

The practical move is to canary a model change before you switch it. Keep the qualified model as an explicit baseline. Run the candidate against a small, representative slice of production-shaped inputs without letting it control the live outcome. Compare both versions against checks that match the workflow. Promote in stages only when the candidate clears a written threshold; otherwise, retain the baseline and investigate the specific failures.

A model name does not always mean “the model you used last month.” Anthropic distinguishes fixed model IDs from convenience aliases for earlier model lines, and says a fixed ID maps to a single model snapshot while an alias can resolve to the most recent dated snapshot.[1] It also notes that serving infrastructure can still cause minor observable behavior changes even when a model ID and weights are fixed.[1] The safe operational conclusion is not to distrust every update; it is to give every behavior-affecting change an observable rollout path.

Define the baseline before you compare it

A useful comparison needs two things: an exact baseline and a result that can be judged. Record the model ID or alias actually resolved, the prompt or workflow version, tool schemas, retrieval configuration, and the evaluation window. Then choose checks appropriate to the job.

For extraction, compare schema validity, required fields, and evidence quotes. For support routing, compare the approved queue, escalation flags, and unsafe guesses. For an agent, compare tool selection, validated arguments, stop outcome, budget use, and whether a proposal changed its external effect. Do not use “the candidate sounds better” as the main acceptance criterion when the workflow has a safer, observable outcome.

Keep the candidate in shadow mode at first. It may receive a scrubbed copy of a real request or a replayed event, but its output cannot send, publish, spend, mutate a record, or replace the baseline response. Store a joinable request ID and only the review data you need. A sensitive transcript copied into a model experiment is not a free source of quality evidence.

Copy this model-change canary card

Use one card per model ID, alias target, provider setting, system prompt, tool-schema, or retrieval change. If several pieces change together, either test them as one named release or isolate them; do not call the result a model comparison when the prompt changed too.

CHANGE
- release_id: [stable ID]
- baseline: [resolved model ID + configuration version]
- candidate: [resolved model ID + configuration version]
- intended workflow: [one named job]
- change scope: [model | alias | provider setting | prompt | tools]

MUST PRESERVE
- input boundary: [redaction, tenant, permissions]
- output contract: [schema / approved labels / citation rule]
- effect policy: candidate runs in shadow; no external actions
- baseline fallback: [exact model/configuration]

EVALUATION
- sample: [representative cases + known breakpoint cases]
- per-case checks: [validity, task rule, evidence, safety, latency/cost]
- reviewers: [code check / human sample / both]
- decision threshold: [for example, no new critical failures and no regression
  on required checks]

ROLLOUT
1. Replay the fixed evaluation set against both versions.
2. Shadow [small percentage or named queue] and log paired receipts.
3. Review failures by category; do not average away a critical miss.
4. Promote to the next percentage only if the threshold passes.
5. On breach, stop promotion and route live work to baseline.

RECEIPT
{release_id, resolved_baseline, resolved_candidate, sample_count,
 passed_checks, failed_checks, critical_failures, decision,
 rollback_target, evidence_refs}

The word resolved matters. Log what the provider actually served, not merely the friendly name configured in an environment variable. An alias may be useful for receiving an upstream improvement deliberately, but it is a poor forensic record if you cannot later tell which behavior handled a disputed request.

Promote by evidence, not an average score

A single average can hide the failure that matters: perhaps the candidate improves summaries yet starts calling a write tool on ambiguous input. Separate hard gates from optimization metrics. Schema violations, cross-scope retrieval, unsupported citations, unapproved external effects, and missed escalation rules should stop a promotion even if style preference or latency improved.

Then expand carefully. Google describes a canary as a progressive rollout that gives a new version to a subset before full release, with defined phases and optional analysis to determine whether to advance.[2] The same structure works for an AI workflow: shadow first, enable the candidate for a small reversible slice, inspect receipts and error categories, then increase exposure. Each stage needs a known fallback that can be selected without asking a model to diagnose itself.

Keep replay cases and live samples distinct. A fixed breakpoint set catches regressions you already understand; shadow traffic finds new request shapes. When the two disagree, inspect the first workflow boundary that changed: retrieval eligibility, prompt assembly, structured output validation, tool arguments, or final generation. “The new model is worse” is not a repair hypothesis.

Failure mode: treating a pinned ID as a permanent quality guarantee

Pinning a model makes a comparison reproducible, but it does not prove that your surrounding system stayed the same. Anthropic explicitly notes that serving infrastructure can change even when model weights and IDs do not.[1] Your tools, prompt renderer, retrieval corpus, safety policy, and provider options can shift too. Conversely, permanently pinning a baseline can leave you exposed to retirement, missed capability improvements, or an untested emergency migration.

Use a pinned baseline for controlled qualification, monitor it in normal operation, and schedule upgrades while you still have time to compare and roll back. The goal is not to freeze an AI system forever. It is to make a change visible, bounded, and reversible before it becomes everyone’s problem.

The habit to keep

treat a model change like a production release. Name the baseline, shadow the candidate, advance only on checks that match the real workflow, and keep a tested route back.

Sources

  1. Anthropic, “Model IDs and versioning”
  2. Google Cloud, “Use a canary deployment strategy”