Make Retrieval Earn the Answer

Retrieval can make an AI answer look careful while leaving one question: do the passages support it? A top result may share the user’s wording but address a different version, exception, or only half the question. If a model must answer after retrieval, it can turn that near-match into a polished conclusion.

The practical move is to put an answerability gate between retrieval and drafting. Before the model writes a substantive answer, require a verdict about the evidence: which parts of the question are directly supported, which are missing or conflicting, and whether the safe next move is to answer, retrieve again, ask a clarifying question, or report that the corpus cannot support a claim.

This is not the same as asking a model to “be grounded.” Self-RAG was proposed in part because indiscriminately retrieving and incorporating a fixed number of passages can lead to unhelpful responses; its framework retrieves on demand and reflects on retrieved passages and generated text.^[1] Corrective Retrieval-Augmented Generation similarly adds a retrieval evaluator that assesses the overall quality of documents for a query and uses its confidence to trigger different retrieval actions.^[2] The operational lesson is simple: retrieval should produce a decision about evidence sufficiency before it produces prose.

Test coverage, not just relevance

A passage can be relevant without being enough. Consider the request: “What is the Enterprise retention period in France, and does it apply to backups?” A result about Enterprise retention in general may be relevant. It is not answerable evidence if it omits France, backups, or both.

Break the request into a few answer requirements before ranking the evidence:

• the object or policy being asked about;
• the scope, such as plan, country, product version, or account type;
• the requested property, rule, or comparison;
• any qualifier that changes the conclusion; and
• the source date or authority required for the decision.

Then make each selected passage earn a specific requirement. “This page is about retention” is not a support statement. “Section 4 names the Enterprise plan and says backups are included” is. This keeps a fluent draft from using one document to imply details it never supplied.

The gate also separates a poor retrieval result from a question that needs more input. If the user has not named a product version, the right response may be a clarification. If the version is known but no eligible source covers it, the right result is a bounded unknown. Neither should be silently repaired by a broad web search or a plausible general rule.

Copy this answerability gate

Run this after hard filters and initial retrieval, but before synthesis or an action proposal. The verdict can be generated by a model, a rule-based checker, or both; the important part is that your application treats it as a decision, not decorative reasoning.

QUESTION: [user's exact question]
AS_OF: [date that the answer must apply]
SCOPE: [product, plan, region, permissions, corpus]

ANSWER REQUIREMENTS
- R1: [named object or policy]
- R2: [scope qualifier]
- R3: [requested fact, decision, or comparison]
- R4: [exception, dependency, or time requirement]

CANDIDATE EVIDENCE
- source_id: [stable ID]
  title/version: [as retrieved]
  locator: [section, page, or chunk]
  excerpt: [bounded passage]

GATE VERDICT
For each requirement, return:
- support: direct | partial | absent | conflicting
- evidence_refs: [source_id + locator]
- limit: [what the cited passage does not establish]

DECISION RULE
- answer: every load-bearing requirement has direct support from an eligible source.
- retrieve_again: a named missing requirement has a safe next query or source.
- clarify: the question lacks a scope or date required to choose evidence safely.
- needs_verification: eligible evidence conflicts, is partial, or no source supports
  a load-bearing requirement.

OUTPUT
{
  "decision": "answer | retrieve_again | clarify | needs_verification",
  "covered_requirements": ["R1", "R2"],
  "missing_or_conflicting": ["R3"],
  "evidence_refs": ["policy-17#backups"],
  "next_safe_action": "[one bounded step]"
}

Notice that partial is not a disguised pass. It is useful information: the corpus may establish the general retention rule but not whether the country-specific exception applies. Preserve that distinction in the user-facing result. “I found a general rule, but not a France-specific source” is more useful—and safer—than an unqualified answer with a citation beside it.

Make the second retrieval targeted

A failed gate should name the gap it is trying to close. If R3 is missing, query for the exact product, scope, and concept rather than rerunning the original broad question with a larger top_k. If two sources conflict, retrieve their versions, effective dates, and authority metadata before asking a model to reconcile them. If the system has no additional authorized corpus, stop. Do not turn “retrieve again” into permission to widen across tenants, private collections, or unreviewed web pages.

Log the verdict with the final answer: requirements, selected source IDs, decision, and any missing condition. That record makes later debugging concrete. A reviewer can see whether a bad answer came from absent evidence, an overly lenient gate, or a generation step that ignored an otherwise adequate evidence set.

Failure mode: making the gate another confident narrator

An answerability check can fail if the same model judges evidence by vague similarity, or if the schema rewards answer more than needs_verification. Treat the gate as a testable component. Include fixtures with a direct answer, a near-match missing a qualifier, conflicting versions, and an underspecified question. Review whether the gate identifies the missing requirement—not merely whether its final label sounds reasonable.

For high-impact answers, combine the gate with authority and time filters, and keep the underlying excerpts available for human review. The gate is not proof that a source is true; it is a disciplined refusal to claim that a source says more than it does.