Pipeline Stages
Classifies intent, detects language, rewrites ambiguous queries, and routes to the appropriate retrieval strategy. Handles multi-turn context injection and sensitive query flagging.
Stage Config
class QueryUnderstanding:
async def process(self, query: str, history: list):
intent = await self.classifier.predict(query)
rewritten = await self.rewriter.rewrite(
query, history, intent
)
strategy = self.router.decide({
"intent": intent,
"has_filters": detect_filters(query),
"needs_live_data": intent in LIVE_INTENTS
})
return QueryPlan(rewritten, intent, strategy){
"action": "QUERY_ANALYSIS_COMPLETE",
"analysis": {
"original": "What is our Q3 procurement policy?",
"rewritten": "Acme Corp procurement policy software licenses Q3 2024",
"intent": "POLICY_LOOKUP",
"entities": {
"timeframe": "Q3 2024",
"department": "Procurement"
},
"routing_strategy": "HYBRID_BROAD_SEARCH",
"pii_detected": false
}
}Platform Services Health
Global Pipeline Stats (24h)
12,847
2.1s
34.7%
98.2%
1.8%
4.2M
RAG fundamentals
RAG (Retrieval-Augmented Generation) solves the fundamental limitations of LLMs: they have a fixed knowledge cutoff, cannot access private data, and hallucinate when forced to recall specific facts. The core idea: instead of asking the model to memorize everything, you retrieve relevant documents at query time and inject them as context. The model becomes a reasoning engine over supplied evidence, not a knowledge store.
The pipeline splits cleanly into an offline indexing phase (run once, or incrementally) and an online query phase (runs per request). Click any box in the diagram to drill deeper into a component.
Offline indexing (top) · Online query pipeline (bottom) · Dashed line = index lookup
Raw sources enter through connectors that parse, normalize, and prepare text for chunking.
You ingest raw sources: PDFs, web pages, SQL tables, markdown docs, APIs. Each source needs a connector that handles parsing (stripping HTML, OCR on scans, handling tables) and normalization into clean text.
Sparse BM25, dense ANN, and hybrid RRF fusion — plus query transforms (HyDE, multi-query, decomposition) applied before the retriever.
Retrieval deep dive
Every retrieval method answers: “how do I measure whether this chunk is relevant to this query?” Sparse and dense methods use completely different notions of similarity, which is exactly why they complement each other.
Sparse retrieval represents both queries and documents as high-dimensional term-frequency vectors — “sparse” because almost all dimensions are zero (most terms do not appear in any given document). The classic algorithm is BM25 (Best Match 25), which improves on raw TF-IDF by saturating term frequency and normalizing for document length.
BM25 score for document D given query Q
score(D, Q) = Σ IDF(qi) · [f(qi, D) · (k1 + 1)] / [f(qi, D) + k1 · (1 - b + b · |D|/avgdl)] IDF(qi) = log((N - df + 0.5) / (df + 0.5))
f(qi, D) = term frequency · |D| = document length · avgdl = average doc length · k1 (typically 1.2–2.0) and b (typically 0.75) are tuning parameters
Mechanically: a BM25 index is an inverted index — a map from term → list of (doc_id, score) pairs. Lookup is O(number of query terms × posting list length), which is fast and exact.
Where sparse excels
Exact keyword matching, product codes (iPhone 15 Pro Max), model numbers, proper nouns, technical identifiers — any case where user vocabulary precisely matches document vocabulary. No semantic gap to cross.
Where it fails
Synonyms, paraphrases, and anything requiring conceptual understanding. “Canine dental care” will not retrieve “dog tooth health” with zero overlap.
Dense retrieval encodes both query and documents as low-dimensional continuous vectors (768–3072 dimensions). Proximity in this space correlates with semantic similarity — “canine dental care” and “dog tooth health” land near each other even with no shared terms.
The architecture is a bi-encoder: pass the query through an encoder to get q_vec, pass each document through the same encoder to get d_vec, and relevance is cosine_similarity(q_vec, d_vec) or dot product. You cannot brute-force compare against millions of vectors per query — this is where ANN (Approximate Nearest Neighbor) indexing comes in.
Where dense excels
Semantic similarity, paraphrase, cross-lingual retrieval, conceptual questions.
Where it struggles
Rare technical terms, product codes, exact strings — the embedding model may map them to similar-looking but meaningless positions in the space.
BM25 and dense retrieval fail in different, mostly non-overlapping ways — so running both and merging is almost always better than either alone. The naive approach — normalize scores from both retrievers and add them — breaks down because BM25 scores (typically 0–30) and cosine similarity (0–1) are on completely different scales.
Reciprocal Rank Fusion sidesteps this by ignoring raw scores and working only on rank positions:
RRF_score(doc) = Σ_retrievers [ 1 / (k + rank(doc)) ] k = 60 (dampens top-rank dominance)
Documents appearing in both lists accumulate scores from both retrievers. Documents in only one list still get credit for that single appearance.
Concrete RRF example (from diagram)
Despite never topping either individual list, doc_A edges out as the hybrid winner. No learned parameters, no normalization — RRF just works, which is why it became the standard.
How to reason through the choice in production system design:
Your corpus has lots of domain-specific jargon, product codes, model numbers, or acronyms that an embedding model has not seen enough of to position well. Also useful when users are expert searchers who know exactly the terminology in the documents.
Users ask questions in natural language that may not match document vocabulary. Cross-language retrieval. Conceptual questions ("what is the company's policy on X") rather than keyword lookups.
Almost always in production — unless you have a very specific reason not to. Tradeoffs are infrastructure complexity (two indexes) and latency (two parallel requests). Most systems accept this because hybrid consistently beats either approach alone on recall benchmarks.
Practical benchmark: on BEIR, hybrid BM25+dense improves nDCG@10 over dense-only by 2–5 points on most datasets — meaningful but not transformative. Gains are largest on technical corpora (TREC-COVID, SciFact) where terminology matters.
Small chunks retrieve precisely but lose context. Large chunks are self-contained but match queries poorly. Parent-child chunking is the standard production solution: embed small child chunks, retrieve parent chunks for the LLM.
Domain-specific models (e.g. medical, legal, code) dramatically outperform general models on in-domain tasks. Always benchmark on your actual data distribution, not MTEB leaderboards alone.
Measure recall@k (does the right chunk appear in top-k?), MRR (how highly is it ranked?), and NDCG. Downstream, measure answer faithfulness (is the answer grounded in context?) and answer relevance (does it address the question?). Tools like RAGAS automate this evaluation.
Query and document do not share vocabulary — dense retrieval misses them.
A retrieved chunk contains misleading information that steers the answer wrong.
Relevant chunk sits in the middle of a long context and the LLM ignores it.
Retrieved chunks are relevant but the LLM ignores them in favor of parametric knowledge.
Agentic RAG adds a routing step where the LLM decides which retrieval strategy to use (or whether to retrieve at all). FLARE and Self-RAG have the model generate in passes, retrieving mid-generation when confidence is low. GraphRAG builds a knowledge graph over the corpus to handle multi-hop reasoning that flat retrieval misses.