A Production RAG Service

Put retrieval behind an API with citations, limits, caching and a fallback.

• Advanced
• 18 min read
• 3 objectives

Before this lessonLesson 14: LangSmith Tracing and Evals

A demo RAG notebook is not a service. Production adds an API, citations, empty-retrieval behaviour, caching and a budget.

The request path

class Ask(BaseModel):
    question: str
    k: int = 4

@app.post("/ask")
def ask(body: Ask):
    hits = store.similarity_search(body.question, k=body.k)
    if not hits:
        return {"answer": "I do not have that in the docs.", "sources": []}
    context = "\n\n".join(h.page_content for h in hits)
    answer = rag.invoke({"question": body.question, "context": context})
    sources = list({h.metadata.get("source") for h in hits})
    return {"answer": answer, "sources": sources}

Guardrails

• Refuse to answer when retrieval is empty or scores are below a threshold.
• Instruct the model: use only the context; say you do not know otherwise.
• Cap max_tokens and timeout the model call.
• Cache embeddings of identical questions (Redis) to cut cost.
• Log thread id, latency, token use, and whether retrieval was empty.

Citations

Show the source path or URL next to the answer. People trust RAG when they can click through; they stop trusting it the first time it invents a policy.