Building a RAG System with LangChain and FastAPI
Retrieval-Augmented Generation lets you ground an LLM in your own data. Here is how to build a production-ready RAG pipeline with LangChain and FastAPI.
RAG (Retrieval-Augmented Generation) is the most practical way to give an LLM access to your private data,without fine-tuning. You store your documents in a vector database, retrieve the most relevant chunks at query time, and inject them into the LLM prompt as context.
RAG architecture overview
- ✓Indexing: chunk your documents → embed them → store in a vector DB (Chroma, Qdrant, Pinecone)
- ✓Retrieval: embed the user query → similarity search → return top-k chunks
- ✓Generation: build a prompt with the retrieved context → call the LLM → return the answer
Setting up the pipeline with LangChain
from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA
# 1. Load and split documents
loader = PyPDFLoader("docs/technical-spec.pdf")
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
docs = splitter.split_documents(loader.load())
# 2. Embed and store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(docs, embeddings, persist_directory="./chroma_db")
# 3. Build the RAG chain
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
)Exposing it via FastAPI
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class QueryRequest(BaseModel):
question: str
@app.post("/ask")
async def ask(body: QueryRequest):
result = qa_chain.invoke({"query": body.question})
return {"answer": result["result"]}Key production concerns: rate-limit your LLM calls, cache frequent queries, monitor token usage per request, and implement hallucination detection (confidence thresholds, source citation).
Chunking strategy matters
The quality of your RAG system depends 70% on chunking and retrieval, and only 30% on the LLM. Experiment with chunk sizes (500–1500 tokens), overlap (10–20%), and retrieval strategies (similarity, MMR, hybrid search) before tweaking the model.
Need help with this topic? AI & RAG Integration
Discover this service →