Modern RAG Architecture: Beyond the Basic Retrieve-and-Generate Loop
A naive RAG pipeline (embed, search, stuff into the prompt) gets you a demo. Production RAG needs a chunking strategy, hybrid retrieval, query transformation, re-ranking and an evaluation loop. Here is the full architecture.
A naive RAG pipeline (embed document chunks, run a similarity search, stuff the top results into a prompt) is enough to build a convincing demo. It is not enough to hold up in production, where retrieval noise, poor chunking and the absence of any feedback loop quietly degrade answer quality. This is the architecture that holds up, component by component.
The pipeline as a whole
Every stage feeds the next, and the evaluation loop feeds back into all of them. A weak stage caps the quality of everything downstream, which is why retrieval fixes usually beat prompt tweaks.
documents
|
[ chunking ] --- fixed / semantic / hierarchical
|
[ embed + index ] --- dense vectors + sparse (BM25)
|
question --> [ query transform ] --- rewrite / HyDE / multi-query
|
[ hybrid retrieval ] --- dense + sparse, fused (RRF)
|
[ re-ranking ] --- cross-encoder on top 20-50
|
[ generation ] --- grounded prompt, cites sources
|
answer ---> [ evaluation ] ---> back to every stageIngestion: chunking is an architecture decision
Chunking strategy determines everything downstream. Fixed-size chunking is the common default, but it cuts sentences and ideas in half. Semantic chunking splits on meaning boundaries; hierarchical chunking keeps small chunks for precise retrieval linked to a larger parent chunk for context. Both outperform fixed-size splitting on real documents.
| Strategy | Retrieval precision | Context kept | Cost to build |
|---|---|---|---|
| Fixed-size | Medium, cuts ideas | Poor | Low |
| Semantic | High | Good | Medium |
| Hierarchical (parent-child) | High | High, parent gives context | Medium to high |
Query transformation: fix the question before you search
The user question is often not the best search query. Rewrite it into a standalone form when it depends on conversation history. Use HyDE (generate a hypothetical answer and embed that instead) when questions and documents use different vocabulary. Fan out a complex question into several sub-queries and retrieve for each. These are cheap LLM calls that lift retrieval quality more than a bigger embedding model.
Hybrid retrieval: vector search alone is not enough
Pure vector similarity is excellent at conceptual matches and weak at exact matches: product codes, error codes, proper names. Combine dense vector search with sparse keyword search (BM25) and merge the two result sets with reciprocal rank fusion.
def hybrid_search(query: str, k: int = 10):
dense_results = vector_store.similarity_search(query, k=20)
sparse_results = bm25_index.search(query, k=20)
return reciprocal_rank_fusion([dense_results, sparse_results], k=k)
def reciprocal_rank_fusion(result_lists, k, rrf_k=60):
scores = {}
for results in result_lists:
for rank, doc in enumerate(results):
scores[doc.id] = scores.get(doc.id, 0) + 1 / (rrf_k + rank + 1)
return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:k]Re-ranking: a second, more expensive pass
Retrieve broadly and cheaply first (top 20 to 50 candidates), then re-rank with a cross-encoder that scores the query and each document together. Cross-encoders are far more accurate than embedding similarity but too slow to run against a whole corpus, so they belong at the re-ranking stage, not the initial retrieval.
Agentic RAG: retrieval as a tool, not a fixed step
Instead of always retrieving once before generating, let the model decide whether retrieval is needed at all, and allow multi-step retrieval for complex questions: search, evaluate the results, search again with a refined query if the first pass was not good enough. It costs more tokens and latency, so reserve it for questions that genuinely need it.
Generation: grounded, and it cites its sources
The prompt instructs the model to answer only from the retrieved context, to say it does not know otherwise, and to attach the source of each claim. Returning the source chunks with the answer is what makes a RAG system auditable, and it is the difference between a plausible answer and a verifiable one.
Evaluation and observability: the part everyone skips
- ✓Measure retrieval quality separately from generation quality: recall@k and precision, not just whether the answer sounded right.
- ✓Log the retrieved chunks alongside every generated answer; you cannot debug a bad answer without seeing what the model actually saw.
- ✓Use a labeled evaluation set or an LLM-as-judge to catch regressions before they reach production.
- ✓Monitor for silent retrieval failures: empty result sets, irrelevant chunks scored as relevant, stale indexes.
| Component | Naive version | Modern version |
|---|---|---|
| Chunking | Fixed size | Semantic or hierarchical |
| Query | Used as-is | Rewritten, HyDE, multi-query |
| Retrieval | Dense top-k | Dense plus sparse, fused with RRF |
| Ranking | Embedding score | Cross-encoder re-rank |
| Control flow | Always retrieve once | Agentic, retrieve when needed |
| Quality | Eyeballed | Retrieval and generation scored separately |
Most RAG quality problems live in retrieval, not generation. Before fine-tuning a model or switching providers, audit what your retriever actually returns for your hardest real-world queries; that is where the fix usually is.
FAQ
- What is hybrid retrieval in RAG?
- Running a dense vector search and a sparse keyword search (BM25) in parallel, then merging the two result lists, usually with reciprocal rank fusion. Vector search handles conceptual matches, keyword search handles exact terms like codes and names, and together they beat either one alone.
- What does re-ranking do?
- It runs a second, more accurate scoring pass over the top 20 to 50 retrieved candidates using a cross-encoder that reads the query and each document together. It is too slow for the whole corpus but cheap on a shortlist, and it noticeably improves which chunks reach the prompt.
- What is agentic RAG?
- A RAG design where retrieval is a tool the model can choose to call, rather than a fixed step. The model decides whether it needs to search, can search multiple times with refined queries, and can evaluate its own results. It suits complex questions and costs more per answer.
- How do I choose a chunking strategy?
- Start with semantic chunking so chunks break on meaning boundaries. Add a parent-child (hierarchical) layer if answers need surrounding context that a small chunk loses. Then tune sizes against your evaluation set rather than guessing.
- How do I measure the quality of a RAG system?
- Score retrieval and generation separately. For retrieval, use recall@k and precision against a labeled set: is the right chunk in the top-k. For generation, check faithfulness to the context and answer relevance, with a labeled set or an LLM-as-judge, and re-run on every pipeline change.
A production-grade RAG architecture is a pipeline with feedback loops, not a single embed-search-generate function. Treat retrieval quality as a first-class, measured metric and the rest of the system becomes far easier to reason about and improve.
Need help with this topic? AI & RAG Integration
Discover this service →