Retrieval-Augmented Generation

Load documents, split, embed, store in a vector database and answer with sources.

A model only knows what it was trained on, and it can confidently make things up. Retrieval-Augmented Generation (RAG) fixes both problems: before answering, your app looks up relevant passages from your own documents and gives them to the model as context. Answers become grounded in your data, and you can show sources.

The pipeline

• Load documents (PDFs, web pages, markdown).
• Split them into chunks small enough to be focused.
• Embed each chunk: convert text into a vector (list of numbers) so that similar meanings are close together.
• Store the vectors in a vector database.
• At question time: embed the question, retrieve the nearest chunks, and generate an answer from them.

pip install langchain-community langchain-text-splitters langchain-chroma langchain-openai pypdf

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

docs = PyPDFLoader("handbook.pdf").load()

splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120)
chunks = splitter.split_documents(docs)

vectorstore = Chroma.from_documents(
    chunks,
    OpenAIEmbeddings(model="text-embedding-3-small"),
    persist_directory="./chroma",
)

chunk_overlap repeats a little text between neighboring chunks so a sentence cut at a boundary is not lost. Chunk size is a key tuning knob: too large and results get noisy, too small and they lose context. Start around 500 to 1000 characters and measure.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

prompt = ChatPromptTemplate.from_template(
    "Answer using ONLY the context. If the answer is not in the context, say you don't know.\n\n"
    "Context:\n{context}\n\nQuestion: {question}"
)

def format_docs(docs):
    return "\n\n".join(f"[{d.metadata.get('page')}] {d.page_content}" for d in docs)

rag = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt | llm | StrOutputParser()
)

print(rag.invoke("How many vacation days do new employees get?"))

docs = retriever.invoke("vacation days")
for d in docs:
    print(d.metadata["source"], "page", d.metadata.get("page"))

Showing where each answer came from builds trust and lets users verify it.

Why RAG systems fail

• Bad retrieval: the right chunk is not in the top k. Improve chunking, add metadata filters, try hybrid (keyword + vector) search or a reranker.
• Bad generation: the model ignores context or invents. Tighten the prompt and require citations.
• Stale data: re-index when documents change.
• Missing evaluation: without a test set of questions you cannot tell if a change helped.

from langchain_community.document_loaders import DirectoryLoader, TextLoader

docs = DirectoryLoader("notes/", glob="**/*.md", loader_cls=TextLoader).load()
chunks = RecursiveCharacterTextSplitter(chunk_size=300, chunk_overlap=50).split_documents(docs)
vs = Chroma.from_documents(chunks, OpenAIEmbeddings(model="text-embedding-3-small"))
for q in ["What is our refund policy?", "Who owns onboarding?"]:
    hits = vs.as_retriever(search_kwargs={"k": 3}).invoke(q)
    print(q, [h.metadata["source"] for h in hits])