Evaluation and Production

Test LLM apps, trace them, control cost and guard against failures.

A demo that worked once is not a product. LLM apps are non-deterministic, depend on outside services and can fail in odd ways, so shipping one needs the same rigor as any system, plus a few new habits: evaluation, tracing, cost control and guardrails.

Build an evaluation set

Collect 30 to 100 realistic questions with the expected answer or key facts. Run your app against them after every change and compare. Without this, every prompt tweak is a guess.

dataset = [
    {"q": "How many vacation days do new hires get?", "must_include": ["15"]},
    {"q": "Who approves expense reports?", "must_include": ["manager"]},
]

def run_eval(chain):
    passed = 0
    for row in dataset:
        answer = chain.invoke(row["q"]).lower()
        ok = all(term in answer for term in row["must_include"])
        passed += ok
        print("PASS" if ok else "FAIL", row["q"])
    print(f"{passed}/{len(dataset)} passed")

LLM-as-judge

For open-ended answers, another model can grade against a rubric (correctness, groundedness in context, tone). Use a strong model for judging, clear criteria and spot-check its grades by hand.

class Grade(BaseModel):
    correct: bool
    reason: str

judge = ChatPromptTemplate.from_template(
    "Question: {q}\nReference: {ref}\nAnswer: {ans}\nIs the answer consistent with the reference?"
) | llm.with_structured_output(Grade)

Tracing

A trace records every step of a run: prompts, retrieved chunks, tool calls, latency, tokens and cost. LangSmith (or open-source alternatives such as Langfuse) capture them so you can see why an answer was wrong.

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="..."

Cost and latency

• Use a smaller, cheaper model for easy steps (classification, extraction) and a stronger one only where needed.
• Trim context: retrieve fewer, better chunks instead of stuffing the window.
• Cache repeated requests, and use provider prompt caching for long shared prefixes.
• Stream responses and run independent steps in parallel.
• Set budgets and alerts on tokens per user and per day.

llm = ChatOpenAI(model="gpt-4o-mini", timeout=30, max_retries=3)
fallback = ChatOpenAI(model="gpt-4o")
safe_llm = llm.with_fallbacks([fallback])

Guardrails

• Validate structured output with schemas and reject or retry bad results.
• Filter or refuse unsafe input and output; log refusals.
• Remove sensitive data (PII) before sending it to third-party models when policies require it.
• Keep a human in the loop for high-stakes decisions.

import time

def run_eval(chain):
    times = []
    for row in dataset:
        t = time.perf_counter()
        chain.invoke(row["q"])
        times.append(time.perf_counter() - t)
    print(f"avg latency: {sum(times) / len(times):.2f}s")
# Run once per prompt variant and compare pass rate and latency.