A naive RAG pipeline,embed document chunks, run a similarity search, stuff the top results into a prompt,is enough to build a convincing demo. It is not enough to hold up in production, where retrieval noise, poor chunking, and the absence of any feedback loop quietly degrade answer quality. This is the architecture that actually holds up.

Ingestion: chunking is an architecture decision, not a detail

Chunking strategy determines everything downstream. Fixed-size chunking is the common default, but it cuts sentences and ideas in half. Semantic chunking (splitting on meaning boundaries) and hierarchical chunking (small chunks for precise retrieval, linked to a larger parent chunk for context) both outperform fixed-size splitting on real-world documents.

Hybrid retrieval: vector search alone is not enough

Pure vector similarity is excellent at conceptual matches and weak at exact matches,product codes, error codes, proper names. Combine dense vector search with sparse keyword search (BM25) and merge the two result sets with reciprocal rank fusion.

def hybrid_search(query: str, k: int = 10):
    dense_results = vector_store.similarity_search(query, k=20)
    sparse_results = bm25_index.search(query, k=20)

    return reciprocal_rank_fusion([dense_results, sparse_results], k=k)

def reciprocal_rank_fusion(result_lists, k, rrf_k=60):
    scores = {}
    for results in result_lists:
        for rank, doc in enumerate(results):
            scores[doc.id] = scores.get(doc.id, 0) + 1 / (rrf_k + rank + 1)

    return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:k]

Re-ranking: a second, more expensive pass

Retrieve broadly and cheaply first (top 20 to 50 candidates), then re-rank with a cross-encoder model that scores the query and each document together. Cross-encoders are far more accurate than embedding similarity, but too slow to run against an entire corpus,so they belong at the re-ranking stage, not the initial retrieval stage.

Agentic RAG: retrieval as a tool, not a fixed step

Instead of always retrieving once before generating, let the model decide whether retrieval is needed at all, and allow multi-step retrieval for complex questions,search, evaluate the results, search again with a refined query if the first pass was not good enough.

Evaluation and observability: the part everyone skips

• ✓Measure retrieval quality separately from generation quality,recall@k and precision, not just "did the answer sound right."
• ✓Log the retrieved chunks alongside every generated answer,you cannot debug a bad answer without seeing what the model actually saw.
• ✓Use a labeled evaluation set or an LLM-as-judge to catch regressions before they reach production.
• ✓Monitor for silent retrieval failures: empty result sets, irrelevant chunks scored as relevant, stale indexes.

A production-grade RAG architecture is a pipeline with feedback loops, not a single embed-search-generate function. Treat retrieval quality as a first-class, measured metric, and the rest of the system becomes far easier to reason about and improve.