Query Rewriting + Hybrid Search + Reranking: From 'Searchable' to 'Accurate' in AI Knowledge Bases

Our RAG knowledge base has been live for half a year. 768-dimensional Embedding vector retrieval, cosine similarity ranking, covering 9 major business domains including customers, orders, products, shipments, production, inventory, and finance. The feature works, and users are using it.

But there's a gap between "works" and "works well."

A salesperson asks "how many containers from last month's orders shipped to the US still haven't arrived at port," and the system's first returned result is a document about US tariff policy. Semantically they're related — both mention "US" and "orders." But the user wants shipping data, not tariff policy.

This kind of mismatch is all too common in vector retrieval. Embedding models understand semantic similarity, not business relevance. Two pieces of text being close in vector space doesn't mean they're the same thing in a business context.

We spent three weeks adding a layer of processing both upstream and downstream of vector retrieval, pushing retrieval accuracy one step further. Three technologies, one goal: make sure what gets searched is what the user wants.

Where the Problem Lies

Before optimizing, we need to understand the three structural flaws of naive RAG.

First, users speak in natural language; retrieval engines want keywords. A salesperson won't type "US export orders container count shipment statistics." They'll say "how many containers from last month's orders shipped to the US still haven't arrived at port." After this sentence goes into the Embedding model, colloquial modifiers like "last month," "how many," and "still haven't arrived at port" dilute the core entity signal. In vector space, this sentence might be very close to a document about "US customs clearance process" — because semantically they do look alike — but it's actually separated from the real shipment statistics data by a layer.

Second, vector retrieval is good at "semantic similarity" but bad at "exact matching." Suppose the knowledge base has a document titled "US Order Shipment Statistics Specification." Vector retrieval might not rank it first — because the semantic distance between the title and the user's colloquial question might not be the smallest. But keyword retrieval (MySQL LIKE or BM25) can match it at a glance. Conversely, keyword retrieval can't understand synonyms and semantic associations. Each method has its blind spots.

Third, high vector similarity in retrieved results doesn't equal high business relevance. Cosine similarity measures directional consistency between two pieces of text in vector space. Two pieces of text being directionally consistent might just mean they discuss the same topic, but their standpoint, scenario, and business meaning could be completely different. The user asks "how to handle claims for delayed delivery," and the system returns a document on "cause analysis of delayed delivery" — similarity 0.87, but it doesn't answer the question.

These three issues aren't flaws of any specific Embedding model; they're structural shortcomings of the naive RAG architecture. The industry already has mature solutions: Query Rewriting addresses input-side issues, Hybrid Search fills blind spots on the retrieval side, and Reranking corrects biases on the ranking side. Combined, these three techniques form an enhancement pipeline of "rewrite → hybrid retrieval → rerank" $TRAE_REF.

Layer One: Query Rewriting — Let the LLM Help Users Speak Clearly

The idea is straightforward: before the user's question goes into vector retrieval, have a lightweight model (we use the lite group model, lowest cost) rewrite the colloquial question into a keyword combination that's more retrieval-friendly.

User says: "how many containers from last month's orders shipped to the US still haven't arrived at port"

After rewriting: "US export orders container count shipment statistics last month export volume"

The rewritten text is then sent to generate Embedding vectors. The effect is twofold: on one hand, it removes colloquial modifiers like "last month," "how many," and "still haven't arrived at port" that don't contribute to retrieval; on the other hand, it adds synonyms and related words like "shipment statistics" and "export volume," broadening the semantic recall scope.

There are several key design decisions in the implementation.

The prompt is the core. Rewrite quality entirely depends on the system prompt design. Our prompt includes five rules: extract core entities (person names, products, countries, document types), add synonyms and related words, remove filler words, output pure text without explanation, preserve the original language. Beyond the rules, we provide two few-shot examples covering both Chinese and English scenarios. Without few-shot, the model would output formats like "rewritten query: XXX" with prefixes that break downstream parsing.

Short queries aren't rewritten. Inputs under 6 characters skip the rewriting step. These queries are typically keywords themselves like "customer" or "order" — rewriting adds no value and only increases latency. This threshold is the inflection point we observed in testing: queries above 6 characters show positive gains after rewriting, while below that the gain is negligible.

5-minute cache. The same query won't repeatedly invoke the model within 5 minutes. We use ConcurrentHashMap to store hash → rewrite result mappings, with automatic timeout cleanup. The cache has a 200-entry limit; when full, expired items are cleared. This design ensures that high-frequency repeated queries (like multiple salespeople querying the same customer simultaneously) don't repeatedly trigger model invocations, keeping costs within acceptable bounds.

Graceful degradation on failure. Model invocation failure, quota exhaustion, network timeout — any exception silently degrades and returns the original query. Rewriting is icing on the cake, not a necessity. The entire RAG flow can't be allowed to fail just because the rewrite service is unavailable.

Layer Two: Hybrid Search — Vector + Keyword, Fused with RRF

This layer solves the problem that "vector retrieval and keyword retrieval each have blind spots."

The approach is to run both retrieval paths in parallel, then fuse the results. Vector retrieval handles semantic recall — understanding synonyms, related concepts, and cross-language matching. Keyword retrieval handles exact matching — raw vocabulary contained in titles and keyword fields. The two paths' results are fused using the Reciprocal Rank Fusion (RRF) algorithm.

RRF Algorithm

RRF is a rank fusion method proposed by Cormack et al. in 2009 $TRAE_REF. The core idea is extremely simple: it doesn't care about the raw scores from each retrieval path (vector similarity vs BM25 scores aren't comparable), only about the ranking each path produces.

Formula:

RRF_score(d) = Σ  1 / (k + rank_i(d))
             i∈lists

Where rank_i(d) is the rank of document d in the i-th retrieval path's results (starting from 0), and k is a smoothing constant (we use 60).

Suppose document A ranks 1st in vector retrieval and 5th in keyword retrieval:

RRF_score(A) = 1/(60+1) + 1/(60+6) = 0.0164 + 0.0152 = 0.0316

Document B ranks 3rd in vector retrieval and 2nd in keyword retrieval:

RRF_score(B) = 1/(60+4) + 1/(60+3) = 0.0156 + 0.0159 = 0.0315

The RRF scores of documents A and B are nearly identical — even though A was much higher than B in vector retrieval. This is the characteristic of RRF: it doesn't favor the absolute score differences of any single retrieval path, it only looks at overall ranking performance. A document that ranks highly in both retrieval paths will have a high RRF score.

Implementation

Our RagEnhancementService class implements the RRF fusion logic. The flow is:

1. Vector retrieval returns top 20 results (sorted by cosine similarity descending)
2. Keyword retrieval returns top 20 results (MySQL LIKE on title/content/keywords fields)
3. Both paths' results go into the reciprocalRankFusion method, k=60
4. After RRF fusion, sort by score descending and take top K

Keyword retrieval implementation is straightforward: tokenize by spaces and punctuation, take words with length ≥ 2, and use MyBatis-Plus LIKE conditions for OR queries. We didn't use MySQL FULLTEXT indexes because our knowledge base isn't large (thousands of entries), and LIKE performance is entirely sufficient. If the knowledge base scales to tens of thousands of entries in the future, we'd consider migrating to BM25 or Elasticsearch.

Choosing k for RRF. The larger k is, the more smoothly rank differences affect the score; the smaller k is, the more advantage top-ranked documents get. We use 60, the default value from the RRF original paper and Elasticsearch $TRAE_REF. In testing, k=60 performed stably, with no need for excessive tuning.

A document that appears in both retrieval paths will have a higher RRF score than one that appears in only one path. This is RRF's natural property and exactly the effect we want — a document that both retrieval paths consider relevant is very likely what the user is looking for.

Layer Three: Reranking — Let the LLM Do the Final Filter

Hybrid Search solves the "can find it" problem. But the order in which found results are displayed is still determined by RRF scores — and RRF scores only reflect ranking, not business relevance.

Back to the earlier example. The user asks "how to handle claims for delayed delivery." The retrieval results might include:

• Document A: "Claim handling process for delayed delivery" (truly relevant)
• Document B: "Cause analysis of delayed delivery" (topically related, but doesn't answer the question)
• Document C: "2024 on-time delivery rate statistics" (data-related, but wrong scenario)

Vector retrieval might rank B above A (because B's wording is closer to the user's question), and keyword retrieval might rank C first (because the word "delivery" matches more times). After RRF fusion, the order might still be B > A > C.

Reranking adds an LLM rerank step before the fused results are returned. The query and candidate documents are sent together to the lite model, which determines which document is most relevant.

Implementation Approach

We have the LLM do a simplified version of pointwise ranking. Specifically:

The candidate documents (default 10) are numbered and sent to the lite model along with the user's query. The prompt asks the model to output the numbered sequence in order of relevance from high to low. For example, given 10 documents as input, the model outputs "2,0,5,1,3,8,4,7,6,9," indicating that document #2 is most relevant and document #9 is least relevant.

We considered using a Cross-Encoder model for reranking — such as BGE-Reranker-v2-m3 from BAAI $TRAE_REF. A Cross-Encoder concatenates the query and document into a single input sequence, sends it to the model, and after joint modeling outputs a relevance score between 0 and 1. The accuracy is indeed higher than LLM prompt ranking.

But we ultimately chose the LLM approach due to deployment cost. A Cross-Encoder requires deploying an additional model service (BGE-Reranker-v2-m3 is about 2.8GB $TRAE_REF), while we're already using the lite model for Query Rewriting, so Reranking reuses the same model API at zero additional deployment cost. For our knowledge base scale (thousands of documents), LLM reranking accuracy is sufficient. If the knowledge base scales to tens of thousands in the future, or if higher accuracy is needed, we can switch to a Cross-Encoder — the interface is already abstracted, just replace the implementation class.

Candidate count control. Reranking only processes the top 10 candidates by default. Sending too many documents to the model makes the prompt longer, increases latency, and degrades accuracy (the model's attention gets diluted). 10 is the balance point between latency and coverage.

Degradation strategy. Like Query Rewriting, when model invocation fails, it silently degrades and returns the original order after RRF fusion. Reranking is a precision optimization, not a functional dependency.

Architecture Design: SDK Layer + SPI Interface

The implementation of the three enhancement techniques isn't placed in the cct-crm-backend business system, but pushed down into our ai-sdk core library. The reason is that these capabilities aren't specific to the foreign trade business — any system using RAG can reuse them.

The architecture has three layers:

┌─────────────────────────────────────────┐
│  cct-crm-backend (宿主系统)              │
│  RagEnhancementProviderImpl              │
│  → callLiteModel (HTTP 调用 LLM API)     │
│  → keywordSearch (MySQL LIKE 检索)        │
│  → getConfig (读取云端配置)               │
└──────────────┬──────────────────────────┘
               │ implements SPI
┌──────────────▼──────────────────────────┐
│  ai-sdk-core (SDK 核心层)               │
│  RagEnhancementService                  │
│  → rewriteQuery (Prompt 构建 + 缓存)     │
│  → reciprocalRankFusion (RRF 算法)      │
│  → hybridSearch (编排: 调 Provider + RRF)│
│  → rerankResults (Prompt 构建 + 解析)     │
│  → RagSearchResult (数据类)              │
└──────────────┬──────────────────────────┘
               │ uses
┌──────────────▼──────────────────────────┐
│  RagEnhancementProvider (SPI 接口)      │
│  → callLiteModel                        │
│  → keywordSearch                        │
│  → getConfig                            │
└─────────────────────────────────────────┘

The core algorithms (RRF, prompt construction, result parsing) are in the SDK layer with zero Spring dependencies. The host system only needs to implement the SPI interface, providing three capabilities: LLM invocation, keyword retrieval, and configuration reading. The benefit of this design is that if other systems integrate with ai-sdk in the future, they can gain full RAG enhancement capabilities just by implementing the Provider interface, without duplicate development.

All configuration switches are in the database. Five configuration items are stored in the cloud pb_ai_config table, with model_group = 'rag_enhancement':

Config Key	Default Value	Description
rag_query_rewrite_enabled	true	Query Rewriting toggle
rag_hybrid_search_enabled	true	Hybrid Search toggle
rag_reranking_enabled	true	Reranking toggle
rag_hybrid_candidate_count	20	Keyword retrieval recall count
rag_rerank_candidate_count	10	Reranking candidate count

Configuration refreshes every 5 minutes; database changes take effect immediately (waiting up to 5 minutes max). Each enhancement feature has an independent switch and can be turned off individually. If a particular enhancement performs poorly in a specific scenario, or cost control is needed, you can disable just that one without affecting the other two.

Results and Costs

After the three enhancement features went live, we observed the following changes in internal testing:

Retrieval accuracy. We prepared 50 sets of test queries (covering five scenarios: customer queries, order queries, product queries, shipment queries, and policy queries), comparing top-1 and top-3 hit rates before and after enhancement. Before enhancement, the top-1 hit rate was about 68%; after, about 82%, a 14 percentage point improvement. The top-3 hit rate improved from 85% to 94%. This data is from internal testing, not a rigorous academic evaluation, but the trend is clear.

Latency increase. With all three enhancement layers enabled, single retrieval latency increased from ~200ms to ~800ms-1.2s (mainly from two LLM invocations). For knowledge base retrieval scenarios, this latency is acceptable — users would rather wait 1 second for a more accurate result than wait 200ms for an irrelevant one. If latency is sensitive, you can turn off Reranking (which contributes about 60% of the additional latency) and keep only Query Rewriting + Hybrid Search, with latency increasing by about 300ms.

Cost. Query Rewriting and Reranking both use the lite model, consuming about 200-500 tokens per invocation. At our API pricing, a single full enhanced retrieval costs about 0.001-0.002 yuan. Combined with the 5-minute cache, high-frequency repeated queries don't repeatedly trigger model invocations, making actual costs even lower.

Future Directions

After landing the three enhancement techniques, RAG accuracy has indeed improved. But the ceiling of naive RAG extends further. As we plan the next steps, we see several directions worth attention.

Cross-Encoder replacing LLM Reranking. Using the lite model for Reranking is currently a cost-driven compromise. If we deploy a dedicated reranking model like BGE-Reranker-v2-m3, accuracy can improve further $TRAE_REF. The advantage of Cross-Encoders is that they jointly encode the query and document, understanding fine-grained relationships between the two layers of text, while the LLM is just doing prompt ranking. The interface is already abstracted; switching is an engineering effort issue, not an architecture issue.

Context Engineering. In 2025, the industry began evolving from RAG to Context Engineering $TRAE_REF. RAG focuses on "what gets retrieved," while Context Engineering focuses on "how to construct the context sent to the model." Beyond retrieval results, it should also integrate multidimensional information like user role, real-time data, conversation history, and permission boundaries, so the model receives not a list of documents but a structured, contextualized work instruction. Our causal model prototype (experience learning + model routing) is actually already doing Context Engineering — injecting experience into the system prompt is context construction. The next step is to more deeply fuse RAG retrieval results with experience injection and model routing.

Multimodal RAG. Foreign trade business has a lot of visually formatted knowledge — product images, inspection report scans, packing list photos $TRAE_REF. Current RAG only handles text; this visual information is either manually entered as text descriptions (with information loss) or completely inaccessible to AI retrieval. Multimodal RAG uses vision-language models to directly embed images, letting AI "understand" product photos and retrieve them. This is highly valuable for our product knowledge base and inspection modules.

Agentic RAG. Current RAG is single-turn retrieval — the user asks once, the system searches once. Agentic RAG lets the AI autonomously decide whether multiple retrievals are needed, cross-domain retrieval, or follow-up retrieval based on preliminary results $TRAE_REF. For example, when a user asks "how does this customer's purchasing trend over the past three years compare to the industry average," the AI might first query customer order data, then query industry benchmark data, then do the comparison analysis itself. This requires Agent architecture support; we haven't deployed Agents yet, but this is a clear direction — when single-turn retrieval can't meet complex query needs, Agentic RAG is the natural evolution path.

Summary

Three technologies, one goal: make vector retrieval go from "searchable" to "accurate."

Query Rewriting solves the input-side problem — the gap between users' colloquial questions and the keyword inputs retrieval engines expect. Hybrid Search solves retrieval-side blind spots — vector retrieval and keyword retrieval each have shortcomings, and RRF fusion lets the two paths' results complement each other. Reranking solves ranking-side bias — high vector similarity doesn't equal high business relevance, and the LLM does the final semantic filter.

With the three layers stacked, retrieval accuracy improved from 68% to 82% (top-1 hit rate), single retrieval cost is about 0.001-0.002 yuan, and latency increase is controllable. All three features have independent switches and can be enabled or disabled as needed.

This enhancement pipeline isn't an endpoint. The next stage of RAG is Context Engineering — from "retrieving documents" to "constructing context," from "returning results" to "delivering answers." We're already on the way.