Run Hybrid Retrieval Before You Blame the Prompt

When an AI answer misses a product code, a policy date, or the exact name of a customer, teams often rewrite the prompt. When it misses a differently worded but clearly relevant passage, they may add more instructions or switch models. Both reactions can be expensive detours when the actual problem occurred earlier: the needed evidence never made it into the model’s context.

A practical repair is hybrid retrieval: run a keyword search and a vector similarity search for the same question, then merge the two ranked result lists before choosing the passages to send to the model. Keyword search is good at exact identifiers, names, dates, quotations, and specialized jargon. Vector search is good at conceptual matches when the question and the source use different wording. Microsoft describes hybrid search as running full-text and vector queries in parallel and merging them with Reciprocal Rank Fusion (RRF).^[1] Elastic similarly recommends RRF to balance documents that rank well in either the keyword or vector route.^[2]

Start with the question shape

Before changing your index, look at the questions people actually ask. A user asking, “What is the EU retention period for Enterprise accounts?” contains several exact handles: a region, a policy term, and a plan name. A keyword route may find the governing clause quickly. A user asking, “How long do we keep data after someone leaves?” may describe the same rule without its formal vocabulary. A vector route has a better chance of finding semantically related wording.

Microsoft notes that keyword search can be especially precise for product codes, specialized jargon, dates, and people’s names, while vector search can find conceptually similar material even without a keyword match.^[1] That is the useful division of labor: exactness and semantic recall, not “old search” versus “AI search.”

Copy this hybrid-retrieval checklist

Use this as a retrieval-stage contract. It is deliberately provider-neutral; adapt the query syntax, filters, and storage fields to your search stack.

INPUT
QUESTION: [user's exact question]
SCOPE: [allowed corpus, tenant, product, version, permissions]

1. APPLY HARD BOUNDARIES FIRST
   - Filter by access, document type, product, locale, and version when known.
   - Never retrieve across a tenant or permission boundary to improve recall.

2. RUN TWO QUERIES IN PARALLEL
   KEYWORD_QUERY: exact names, IDs, error codes, quoted phrases, and key terms.
   VECTOR_QUERY: an embedding of the full natural-language question.
   - Retrieve the same candidate depth from each route (start with 20–50).

3. FUSE BY RANK, NOT RAW SCORE
   - Use Reciprocal Rank Fusion or another documented rank-fusion method.
   - Keep route metadata: keyword_rank, vector_rank, fused_rank.
   - Deduplicate by stable chunk or document ID.

4. SELECT EVIDENCE, NOT JUST RESULTS
   - Read the top fused passages.
   - Keep only passages that directly support the answer or necessary caveat.
   - Preserve source URL, title, version/date, and chunk location.

5. LOG THE DECISION
   - Question, filters, candidates from each route, final passages, answer outcome.
   - Tag misses: absent source, bad chunk, bad ranking, or answer-generation error.

6. EVALUATE BEFORE TUNING
   - Test exact-ID questions, paraphrased questions, and mixed questions separately.
   - Compare keyword-only, vector-only, and hybrid on the same labeled set.

The logging step matters because hybrid retrieval gives you a diagnostic split. If the right passage appears in the fused top results but the answer ignores it, improve the answer prompt, citation instruction, or context layout. If it appears only at rank 80, work on ranking or candidate depth. If neither route finds it, the problem is likely indexing, chunking, permissions, or missing source content. Those are different repairs; a stronger “be accurate” prompt does not solve all of them.

Keep scores in their lane

Do not add BM25-style keyword scores directly to vector similarity scores unless your search system explicitly documents how to calibrate them. They are produced by different ranking functions and often have incompatible ranges. Rank fusion avoids pretending that a score of 7.2 from one system means the same thing as 7.2 from another.

Azure’s hybrid-search documentation describes the two query types as using different ranking functions and returning one unified result set through RRF.^[1] The practical implication is to retain the per-route ranks in your logs even when the user sees only one answer. When a policy clause surfaces because it was keyword rank 1 but vector rank 43, that is useful evidence about your corpus and queries.

Also keep metadata filters separate from relevance. Permissions, tenant IDs, document lifecycle, product version, and locale are boundaries, not hints. Apply them before or within both retrieval routes. A semantically perfect passage from the wrong customer, outdated release, or inaccessible document is not a helpful candidate.

Failure mode: a wider net that dilutes the evidence

Hybrid retrieval can make an answer worse if it increases the number of loosely related passages without preserving a disciplined selection step. A generic question may return popular keyword matches and semantically adjacent chunks that disagree, refer to different versions, or bury the governing policy. More retrieved material can distract a model just as surely as missing material can.

Start with a small labeled set of real questions. For each question, mark the passage or document that must be found, then compare keyword-only, vector-only, and hybrid results. Inspect failures by route and by corpus version. Use reranking or narrower filters only after you know whether the problem is recall, relevance, or evidence selection.