LangSmith Tracing and Evals

Trace every run, build a dataset and score answers with evaluators.

• Advanced
• 16 min read
• 3 objectives

Before this lessonLesson 13: Multi-Agent Patterns

LangSmith records inputs, outputs, latency and token counts for every runnable. You cannot improve what you cannot see.

Tracing

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=lsv2_...
export LANGCHAIN_PROJECT=support-bot

Runs appear in the project. Click a trace to see the prompt after variables were filled, the retrieved chunks, and the model output.

Datasets and evaluators

from langsmith import Client
client = Client()
ds = client.create_dataset("refund-questions")
client.create_examples(
    dataset_id=ds.id,
    inputs=[{"question": "How long do refunds take?"}],
    outputs=[{"answer": "Within 5 business days."}],
)

An evaluator can be a string check, embedding similarity, or another model judging faithfulness to retrieved context. Run it in CI on a small golden set so a prompt change cannot silently regress.