Hybrid RAG for enterprise: when should lexical and semantic search be combined?
Lexical search retrieves exact identifiers and phrases. Semantic search handles paraphrases. Hybrid RAG combines both, but adds indexing, fusion and evaluation work that teams must control.
In a retrieval-augmented generation (RAG) system, the model can only answer reliably when the retriever supplies the right passages. Lexical search, often based on BM25, is strong on exact terms. Semantic search, based on vectors, connects paraphrases and related concepts. Hybrid RAG runs both searches and fuses their candidates. It fits corpora that mix jargon, identifiers, synonyms and varied question wording. It is not automatically better in every context.
The short answer
Start with hybrid RAG when users search for codes, versions, names or exact references as well as concepts expressed through synonyms. Run BM25 and dense retrieval in parallel, fuse ranks with a method such as Reciprocal Rank Fusion (RRF), then evaluate retrieval by query family. Keep lexical-only search when vocabulary is stable and term precision dominates. Keep dense-only search when queries are highly paraphrased and exact identifiers are not critical.
Nexxom diagram. The two retrievers produce independent candidates. Fusion and reranking happen before passages are selected for the model.
Lexical, semantic or hybrid search: what is the difference?
Lexical search
Lexical search compares query terms with document terms. BM25 considers factors such as term frequency and document length. This approach fits error codes, contract references, version numbers, product names and regulatory phrases. It is also inspectable: a team can review which terms contributed to a result.
Its limitation is familiar. A query that says “deferred refund” may not retrieve a passage saying “restitution after a waiting period” if the corpus contains neither the same terms nor a lexical expansion.
Semantic search
Semantic search turns queries and passages into vector representations. It can connect formulations that share an intent even when their words differ. Dense Passage Retrieval research demonstrated the value of dense representations for some open-domain question-answering tasks. An independent replication also showed why the corpus, protocol and lexical baseline still matter.
Vector search can miss a rare identifier, an exact string or a version. A vector expresses semantic proximity, not a character-for-character match guarantee.
Hybrid search
Hybrid search runs lexical and vector queries against the same corpus, then joins their result lists. Azure, OpenSearch and Pinecone documentation describe this design as a way to cover complementary failure modes. The effect still needs to be tested on the target corpus: two retrievers that return the same passages add cost without adding coverage.
| Query type | Lexical | Semantic | Hybrid |
|---|---|---|---|
| Exact error code or SKU | Very suitable | Risk of an incorrect semantic neighbour | Suitable when the exact term stays primary |
| Paraphrased question | May miss the wording | Very suitable | Suitable when the corpus also contains names |
| Contract or standard name | Suitable, especially with weighted fields | May return nearby documents | Suitable with lexical filtering and controlled fusion |
| Short ambiguous query | Easy to diagnose | May overinterpret intent | Useful only with reranking and ambiguity tests |
| Small homogeneous corpus | Often enough | May be unnecessary | Extra cost must be justified |
When hybrid RAG creates real value
Corpora mix natural language and identifiers
A support knowledge base may contain free-form descriptions, version numbers and error messages. Real queries alternate between “how do I enable synchronisation?” and “error E-1042 on v4.8”. Dense retrieval helps with the first wording. BM25 protects the second.
Users use different synonyms from the documents
Business, support and legal teams do not always use the same vocabulary. Semantic search may retrieve a relevant passage despite the paraphrase. The lexical branch keeps a visible signal for important and mandatory terms.
Traceability matters as much as coverage
An enterprise RAG system must explain why a passage was selected. The lexical branch often supplies an inspectable signal. The dense branch can expand coverage. The final decision should retain ranks, retriever identifiers, index version and the passage sent to the model.
When not to add a second branch
Hybrid pipelines are not free insurance. They may double indexes, embeddings, search calls and consistency checks. A corpus of short procedures with stable terms and queries close to titles may be well served by a carefully configured lexical engine. A narrative corpus with few identifiers may start with dense retrieval.
Do not choose hybrid search to compensate for poor chunking, missing metadata or stale documents. A second retriever does not repair a badly classified source. Fix quality, freshness and passage structure first.
How to build the hybrid pipeline
1. Index the same knowledge units
The document, passage, metadata and permissions must be consistent across both branches. If BM25 searches a full document while the vector represents a different fragment, fusion may favour results that are not comparable. Keep a stable passage identifier and apply access filters before context reaches the model.
2. Retrieve more candidates than you need
Each retriever should produce a candidate list. The exact size depends on corpus scale, cost and noise. Determine it through evaluation rather than copying a constant from an example. A reranker can analyse the fused list, but it cannot retrieve a document absent from both initial lists.
3. Fuse ranks, not incomparable scores
BM25 scores and vector similarity usually have different scales. Azure documentation recommends Reciprocal Rank Fusion to combine the ranks from multiple queries. RRF contributes according to a result's position in each list. It reduces the need to calibrate raw scores, but its parameters and candidate counts still need testing.
4. Add a reranker only when the gain pays for the cost
A reranker examines the relationship between the query and candidates more deeply. It can improve context precision, but adds latency and compute cost. Measure its effect on difficult queries, not only on the average. If it often changes the first result, inspect chunking and metadata before adding another layer.
5. Log the retrieval decision
Keep the normalized query, filters, index versions, candidates from each branch, fusion, reranking, final context and answer. These traces distinguish a missing document from a ranking error or a generation error. Security and access controls must be evaluated before fusion, not after context reaches the model.
What evaluation protocol should be used?
Public benchmarks such as BEIR help explain retriever families, but they do not replace a query set representing the enterprise. Build annotated queries across categories: exact terms, paraphrases, multi-step questions, names, versions, no-answer queries and content subject to access rights.
| Metric | Question | Why it matters |
|---|---|---|
| Recall@k | Does the useful passage appear in the first k results? | Detects evidence lost before generation |
| Precision@k | How many candidates are actually useful? | Measures noise passed to reranking and the model |
| nDCG@k or MRR | Is the useful passage near the top? | Compares ranking quality, not only presence |
| Grounded-answer rate | Does the answer remain supported by the passages? | Connects retrieval and generation |
| Latency and cost | What does each query cost? | Prevents an unusable relevance gain |
| Access and freshness | Does the result respect rights and the latest version? | Avoids relevant but unauthorized or stale answers |
Compare at least three configurations on the same query set: lexical-only, dense-only and hybrid. Record embedding versions, chunking parameters, list sizes, fusion method and thresholds. Without this protocol, an average gain can hide a regression on regulated queries or identifiers.
Decision matrix for a product team
| Corpus and query profile | Recommended starting point | Signal to move to hybrid |
|---|---|---|
| Short procedures, stable vocabulary, many codes | Lexical with explicit fields and filters | Paraphrases regularly miss the useful passage |
| Natural questions, few identifiers, narrative corpus | Dense with coverage evaluation | Required terms or names are lost |
| Technical docs, tickets, versions and synonyms | Hybrid with RRF and metadata | The two branches return complementary sets |
| Highly sensitive corpus with fine-grained rights | Choose the most controllable engine | Hybrid is acceptable only with identical tested filters |
| Strict latency and small volume | Simplest pipeline above the quality threshold | A measured gain pays for the second branch |
Common mistakes
- Combining BM25 and vector scores without normalization or a documented method.
- Applying different access filters in the two branches.
- Sending stale passages to the model because they are semantically close.
- Increasing candidate counts without measuring noise and latency.
- Adding a reranker before creating an annotated query set.
- Comparing systems with different embeddings, chunks or index versions.
- Confusing a good passage ranking with a correct, grounded answer.
Frequently asked questions
Is hybrid RAG always better than dense retrieval?
No. It can cover complementary failures, but its value depends on the corpus, queries, filters, chunk quality and latency budget. Measure all three configurations on your own queries.
Does RRF replace a reranker?
No. RRF fuses ranks from first-stage lists. A reranker then re-evaluates the query and candidates. It may improve precision, but adds cost and cannot recover a passage missing from the initial lists.
Do we need two databases?
Not necessarily. Several engines store lexical and vector indexes in one collection. The choice depends on security, operations, cost, migration and available features. Identifiers and filters must remain consistent.
Which metric should we measure first?
Start with Recall@k to check that useful evidence is retrieved, then measure ranking and grounded answers. Add latency, cost, freshness and access compliance for the production decision.
Conclusion
Lexical and semantic search answer different retrieval needs. BM25 protects exact terms, identifiers and inspectability. Dense retrieval connects paraphrases and intent. Hybrid RAG is justified when those strengths are genuinely complementary. Build an enterprise query set, compare lexical, dense and hybrid retrieval, fuse ranks with an explicit method, and verify relevance, grounding, latency, cost and access. Simplicity remains the best choice until a second branch delivers a measured gain.
For the next decision, read our guide to choosing an AI model for business and our method for selecting an agentic process.
Primary sources verified on August 11, 2026
- Microsoft Learn, Hybrid search in Azure DocumentDB
- Microsoft Learn, Hybrid search scoring and Reciprocal Rank Fusion
- OpenSearch documentation, Hybrid query
- Pinecone documentation, Hybrid search
- BEIR, A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Dense Passage Retrieval for Open-Domain Question Answering
- A Replication Study of Dense Passage Retriever
- HYRR, Hybrid Infused Reranking for Passage Retrieval
Performance depends on data, model versions, chunking and index parameters. No public result replaces an evaluation on your enterprise queries and access rules.
