Vector-only RAG fails in production on the very queries your users care about most: SKUs, ticket IDs, error codes, and exact policy clauses.
The fix is not “better embeddings”. The fix is treating retrieval like a search system again: hybrid candidate generation (sparse + dense), a real reranker, and access-control-aware filtering that runs before any model sees protected text.
Where vector-only RAG actually breaks
Dense retrieval is good at “aboutness”. It is not reliably good at “this exact string”, and it gets worse as your corpus grows and your content becomes more operational (support tickets, changelogs, policies, catalogues).
The failure modes below are not theoretical. They show up repeatedly in practitioner write-ups and shared task systems that end up converging on sparse+dense fusion plus reranking. The TREC RAG 2025 track submissions, for example, include hybrid retrieval with BM25/SPLADE + dense embeddings and LLM reranking as an explicit design choice, not a novelty add-on. (trec.nist.gov)
1) Identifier queries: SKUs, incident IDs, order numbers
If the user types something like:
- “ARG-2024-1847”
- “SKU 00-19384-B”
- “E_UNAUTHENTICATED 16”
…they do not want semantic neighbours. They want the one record that contains that token, and they want it even if the surrounding text is boilerplate.
Dense embeddings routinely miss or under-rank these cases because:
- Tokenisation can split IDs in odd ways (hyphens, slashes, mixed case).
- The model learns that rare strings are not meaningful features.
- Similarity is dominated by surrounding context (“error”, “failed”, “login”), not the discriminative identifier.
A practical example of this class of issue appears in a public “production-grade hybrid RAG” implementation where “Find document ARG-2024-1847” is explicitly called out as failing on vector-only and succeeding with hybrid retrieval plus fusion and reranking. (github.com)
2) Exact-phrase needs: policy clauses, legal text, compliance language
Policy and legal content is often repetitive. The “meaning” of adjacent clauses is similar, while the exact wording matters.
Dense similarity is prone to retrieving:
- a nearby clause (“close enough”),
- an older revision,
- or a summary paragraph that sounds right but is not the actual text you must quote.
This is one reason compliance-focused “policy-first” hybrid RAG architectures emphasise lexical retrieval and access control as first-class design constraints, not optional optimisations. (zenodo.org)
3) Negation and exception handling: “X is allowed, except when Y”
Embedding retrieval tends to “blur” exception language:
- “You can refund within 30 days” and “You cannot refund after 30 days” are semantically close.
- “Covered” and “not covered” can sit in similar neighbourhoods.
Hybrid candidate generation helps, but this is where reranking earns its keep: a model that scores query+passage jointly (cross-encoder) can learn that the exception clause is the one that answers the question.
4) Retrieval degrades silently at scale
Teams often discover retrieval regressions only after complaints, because the system still returns something plausible.
A particularly nasty version: you “add hybrid” but then accidentally gate on vector similarity after fusion, throwing away lexical-only hits. A practitioner write-up describes exactly this failure mode with Reciprocal Rank Fusion (RRF): because RRF intentionally discards absolute scores, a post-fusion similarity threshold can erase BM25-only results and make your system behave as vector-only without anyone noticing. (reddit.com)
Hybrid retrieval, in one sentence
Hybrid retrieval means you generate candidates using both sparse lexical search (BM25/SPLADE-style) and dense vector search, then fuse and rerank to get high recall and high precision.
This is also what you see in competitive RAG retrieval settings: teams combining BM25/SPLADE with dense models and then reranking with LLMs or neural rerankers in TREC RAG 2025. (trec.nist.gov)
A production-ready hybrid recipe (what to build)
You can implement this in most modern stacks (OpenSearch/Elasticsearch + a vector DB, Postgres + pgvector + a BM25 service, Vespa, etc.). The important part is the shape of the pipeline and where you put the guardrails.
Stage 0: normalise, enrich, and chunk for retrieval (not for embeddings)
Do not start by choosing an embedding model. Start by deciding what a retrievable “unit” is.
- Chunk by semantics and structure: headings, sections, policy clauses, error catalog entries.
- Preserve exact strings in the chunk text (IDs, codes, SKU tokens) even if you also store a normalised form.
- Attach metadata you will actually filter on: tenant, department, product line, document version, effective date, ACL principals.
If you cannot answer “what version of this policy is in the index?” you will ship confident wrong answers.
Stage 1: ACL-aware filtering before retrieval (or at least before reranking)
If the user is not allowed to see a document, the model must not see it either.
In practice this means:
- Apply access control filters at query time for both sparse and dense retrieval.
- If your vector store cannot filter efficiently, don’t pretend. Put vectors behind a service that can.
- Log the effective filter set used for each query (for audits and debugging).
This design is explicitly highlighted in “policy-first” enterprise hybrid RAG architectures, which enforce access control before reranking or generation. (zenodo.org)
Stage 2: candidate generation with two retrievers
Run both arms in parallel:
- Sparse: BM25 (or SPLADE if you can run it) for lexical recall.
- Dense: a bi-encoder embedding model for semantic recall.
Pull enough candidates to give the reranker room to work. Typical patterns:
- top 50–200 from BM25
- top 50–200 from dense
The right number is an engineering trade-off: more candidates improves recall but costs latency and reranking compute.
Stage 3: fuse rankings (don’t average scores)
Dense scores and BM25 scores are not commensurate. Treating them as if they are often produces brittle behaviour across corpora.
A simple, robust default is Reciprocal Rank Fusion (RRF):
- It fuses based on rank positions, not raw scores.
- It is hard to “break” with scale changes in either arm.
This is also common in hybrid retrieval write-ups and shared task systems; for instance, TREC RAG runs explicitly reference multi-method hybrid pipelines and fusion strategies. (trec.nist.gov)
Stage 4: rerank with a model that actually reads the text
Hybrid gets you recall. Reranking gets you precision.
Two common production options:
- Cross-encoder reranker (BERT-style): highest precision, higher cost.
- Late-interaction reranker (ColBERT-style): better trade-off between quality and throughput; used widely enough to have continuing research and optimisation. (arxiv.org)
The key point: reranking must be fed the full query and the full candidate passage. That joint scoring is what bi-encoders give up.
Stage 5: build an “evidence pack”, then generate
Generation should be downstream of retrieval, not intertwined with it.
- Select top N reranked chunks (often 3–8).
- Include citations/IDs for each chunk.
- Make the generator answer only from those chunks.
If you cannot show the evidence, you cannot debug the system when it is wrong.
How to prove the lift (without fake benchmarks)
Do not ship hybrid on vibes. Ship it because it demonstrably fixes your known failure modes.
A lightweight evaluation loop looks like this:
- Build a golden set of 50–200 queries from real logs.
- Label the “must retrieve” document/chunk for each query.
- Track retrieval metrics: - Recall@K (did we retrieve the right chunk anywhere in top K?) - MRR (did we rank it near the top?)
- Track answer metrics: - groundedness (answer cites retrieved evidence) - refusal correctness (when there is no evidence) - policy/version correctness (did it cite the right revision?)
You can also use an LLM-as-judge setup for answer grading as long as you keep it bounded (judge against the evidence pack, not against the open web) and spot-check disagreements.
Academic and shared-task work increasingly evaluates hybrid pipelines as a combined system (retrieve + rerank) rather than only comparing embedding models. For example, a 2026 benchmark on financial text-and-table documents reports that two-stage hybrid retrieval plus neural reranking outperforms single-stage methods, and also notes that BM25 can beat dense retrieval on precision-heavy domains. (arxiv.org)
Trade-offs and “do not do this” notes
Hybrid is not free. It is, however, usually cheaper than being wrong at scale.
What you pay
- Operational complexity: two indexes (sparse + dense), plus reranker infra.
- Latency: especially with cross-encoder rerankers.
- More moving parts to observe: per-arm contribution, fusion behaviour, and ACL filters.
What you must not do
- Do not apply a single similarity threshold after fusion and assume it is meaningful across arms (it is not). (reddit.com)
- Do not rerank content the user cannot access (filter first).
- Do not “chunk smaller” to fix retrieval. Too-small chunks often remove the discriminative tokens and make reranking harder.
- Do not evaluate only on “nice” semantic questions. Your system lives or dies on the nasty ones: IDs, versioned policies, and exception clauses.
Where codeversols fits (briefly)
If you are building this in-house, you want an engineering team that treats retrieval, access control, and evaluation as one system. That is the work: integrating sparse+dense retrieval, implementing ACL-aware filtering properly, and shipping an eval harness that catches regressions before customers do.
At codeversols we do the unglamorous parts: production pipelines, observability, and the guardrails around RAG so it behaves like a dependable system rather than a demo.
Close
Vector-only RAG fails in predictable, repeatable ways: identifiers, exact clauses, and scale-driven regressions that look like “the model got worse”.
Hybrid retrieval with reranking and ACL-aware filtering is no longer an advanced option; it is the baseline architecture that survives contact with real corpora. Build it with an evaluation loop from day one, and you will spend your time improving coverage instead of arguing with retrieval ghosts.
