How to Build Regression Test Datasets for AI Agents from Production Failures
To build regression tests from production failures, run one loop continuously: trace every agent run, flag the runs that went wrong (user feedback, online evaluator scores, errors), turn each confirmed failure into a dataset case with the original input, recorded tool responses and a grader, run that dataset in CI as a merge gate, and watch each release on a canary with online evals, rolling back when the score drops.
The tooling answer is just as short. LangSmith, Langfuse, Braintrust, Arize Phoenix, W&B Weave, Opik and MLflow all ingest traces and can turn a trace into a dataset item, and most of them score live traffic and alert when a score falls. We found no documented automatic rollback of a release in any of them. That job belongs to the release layer: a feature flag with guarded rollouts, a canary controller such as Argo Rollouts, or a small script that switches your model-routing config back to the stable version. All three are covered below, with working code.
Our AI agent testing and evaluation guide covers golden datasets and trace replay in general; this post goes deep on the loop from a production failure to a test that keeps it fixed.
The loop at a glance
| Step | What you do | What you end up with |
|---|---|---|
| 1. Capture | Trace every run: input, versions, tool calls and results, output | A trace you can replay |
| 2. Triage | Route bad feedback, low judge scores and errors to review; label the failure mode | Confirmed, labelled failures |
| 3. Promote | Copy the failure into a dataset with tool fixtures, expected behavior and a grader | A regression case linked to its trace |
| 4. Gate | Run the dataset on every pull request | A merge that cannot bring the bug back |
| 5. Watch | Canary the new version, score it live, roll back if it is worse | Bad releases that undo themselves, and new failures for step 2 |
1. Capture traces you can replay
A failure you cannot reproduce cannot become a test. A useful trace holds everything needed to rerun the turn without touching production systems:
- The full input: the user message, conversation history and any state the agent read.
- Versions: model ID, prompt version and a release tag (step 5 needs it).
- Every tool call: name, arguments and exact response. These become your test fixtures.
- Retrieved documents, so you can tell a retrieval miss from a reasoning error (see how to evaluate RAG retrieval).
- Outcome signals: errors, tokens, latency, and a session ID to attach feedback to.
Use OpenTelemetry where you can. The OpenTelemetry GenAI semantic conventions now live in their own repository and are still marked Development, so names can change. They define agent operations such as invoke_agent and execute_tool, and attributes such as gen_ai.operation.name, gen_ai.agent.name, gen_ai.conversation.id, gen_ai.request.model and gen_ai.usage.input_tokens. Message content such as gen_ai.output.messages is opt-in because it can hold personal data, so decide deliberately what you record (our GDPR guide for AI agents covers retention). All seven platforms compared below accept OTLP traces, so instrumenting once keeps you free to switch.
LangChain tracing in practice
| Destination | How LangChain and LangGraph runs get there |
|---|---|
| LangSmith | Set LANGSMITH_TRACING=true and LANGSMITH_API_KEY; add LANGSMITH_PROJECT in project-based workspaces. No code changes. LANGSMITH_OTEL_ENABLED=true switches export to OpenTelemetry. |
| Langfuse | from langfuse.langchain import CallbackHandler, then pass the handler in config={"callbacks": [handler]} when you call invoke. |
| Arize Phoenix | Install openinference-instrumentation-langchain and call register() from phoenix.otel with auto-instrumentation, or use LangChainInstrumentor. |
In serverless functions, flush before exit (LangSmith’s docs suggest synchronous callbacks or wait_for_all_tracers() in Python), or you lose the traces of timeouts and crashes, the failures you most need.
2. Triage: find the failures worth keeping
Most traces are fine. Combine several signals to surface the ones that are not; each signal misses different failures:
| Signal | How to capture it | What it misses |
|---|---|---|
| Explicit feedback | Thumbs down or a rating, logged as a score on the trace | Most users never click |
| Implicit feedback | User rephrases the request, asks for a human, abandons the task, or rewrites the agent’s draft | Needs product events joined to traces |
| Online evaluators | LLM judge or code check on a sample of live traces (groundedness, policy, task completed) | Judge errors; cost limits sampling |
| Error signals | Tool exceptions, error.type on spans, timeouts, step-limit hits, schema failures | Confident wrong answers raise no error |
| Outliers | Runs with far more steps, tokens or cost than the median for that task | Short wrong answers |
Automate the routing, not the judgment. A LangSmith automation rule, for example, takes a filter (negative feedback, a low evaluator score, an error), a sampling rate and an action such as add to annotation queue or add to dataset; the other platforms offer similar rules or review queues. Someone then reviews the queue on a schedule, confirms each failure and tags its mode:
- wrong_tool or missing_action: wrong tool, or claimed an action it never took
- bad_args: right tool, wrong amount, ID or date
- unsupported_claim: not backed by retrieved context or tool output
- retrieval_miss: the right document never came back
- policy: should have refused or escalated
- format: broke the schema or required structure
- loop: repeated a call until the step limit
Counting failures by mode shows where to spend effort, and near-identical failures become one case, not twenty.
3. Turn a failure into a dataset case
Say a customer wrote “I was charged twice for order 8812, please refund one” and the agent cancelled the order instead. Here is that failure as a regression case. In the file it is one JSON object per line (JSONL); it is spread out here for reading:
{
"id": "dup-charge-refund-8812",
"source_trace": "trace_7f3a9c",
"failure_mode": "wrong_tool",
"input": {"messages": [{"role": "user", "content": "I was charged twice for order 8812, please refund one"}]},
"fixtures": {"get_order": {"id": "8812", "charges": [49.0, 49.0]}},
"expect": {
"tools_called": ["get_order", "create_refund"],
"tools_not_called": ["cancel_order"],
"args": {"create_refund": {"amount": 49.0}},
"output_contains": ["refund"]
},
"trials": 3,
"min_pass": 3
}- Reproduce first. Run the case on the current version and confirm it fails. If it passes, you missed some context or the grader is too loose.
- Freeze the world with fixtures. Copy the tool responses from the trace into
fixturesand have your harness return them instead of calling live APIs, so the test checks the agent’s decisions, not today’s database. - Write the expectation as behavior. “Calls
create_refundfor 49.0 and nevercancel_order” survives a prompt rewrite; a golden output string does not. - Pick the cheapest grader that catches the failure (table below).
- Keep provenance: source trace, failure mode, date and the version that failed.
- Strip personal data before the case lands where developers and CI can read it.
| Failure mode | Grader |
|---|---|
| wrong_tool, missing_action, loop | Check the tool-call sequence and count |
| bad_args | Compare arguments to expected values |
| format | JSON schema or regex validation |
| retrieval_miss | Expected document ID in the top k results |
| unsupported_claim, tone, policy wording | LLM judge with a written rubric, calibrated on human labels |
Deterministic graders are free, fast and consistent. Use an LLM judge only where meaning matters, give it a rubric with pass and fail examples, and check it against a few dozen human labels before it can block a merge (Langfuse’s score analytics, for one, shows judge-versus-human agreement). Every platform in the comparison below can create the dataset item from a trace for you; you still add the expectation and the grader.
4. Run the dataset in CI as a regression gate
A golden set gates on an overall pass rate. A case from a real incident should be must-pass: the bug was fixed, so any failure means it is back. Agents are not deterministic, so each case runs several times and has its own minimum.
This pytest runner runs each case trials times with its fixtures, applies the checks in expect, and fails the build if a case passes fewer than min_pass times. Your run_agent must return the final output and the list of tool calls.
# evals/test_regressions.py
import json
from pathlib import Path
import pytest
from my_agent import run_agent # returns {"output": str, "tool_calls": [{"name": str, "args": dict}]}
DATASET = Path(__file__).with_name("regressions.jsonl")
CASES = [json.loads(line) for line in DATASET.read_text().splitlines() if line.strip()]
def grade(case, result):
"""Return a list of failure reasons. An empty list means the run passed."""
expect, errors = case["expect"], []
called = [c["name"] for c in result["tool_calls"]]
remaining = iter(called) # expected tools must appear in this order; extra calls are allowed
if not all(name in remaining for name in expect.get("tools_called", [])):
errors.append(f"expected tools {expect['tools_called']} in order, got {called}")
for name in expect.get("tools_not_called", []):
if name in called:
errors.append(f"called forbidden tool {name}")
for name, want in expect.get("args", {}).items():
got = next((c["args"] for c in result["tool_calls"] if c["name"] == name), {})
for key, value in want.items():
if got.get(key) != value:
errors.append(f"{name}.{key}: expected {value!r}, got {got.get(key)!r}")
text = result["output"].lower()
errors += [f"output missing {s!r}" for s in expect.get("output_contains", []) if s.lower() not in text]
errors += [f"output contains {s!r}" for s in expect.get("output_not_contains", []) if s.lower() in text]
return errors
@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_production_regression(case):
trials, min_pass = case.get("trials", 1), case.get("min_pass", 1)
failures = []
for _ in range(trials):
result = run_agent(case["input"], fixtures=case.get("fixtures", {}))
errors = grade(case, result)
if errors:
failures.append(errors)
passed = trials - len(failures)
assert passed >= min_pass, f"passed {passed}/{trials}, first failure: {failures[0]}"Run it with python -m pytest evals/ from the project root. A failure reads like passed 0/3, first failure: ['called forbidden tool cancel_order'] under the case ID, so the CI log points straight at the incident.
- Set
min_passper case: 3 of 3 for a refund, perhaps 4 of 5 for a wording issue. - Keep two gates: must-pass incident cases, plus your golden set’s pass rate against the main branch (see the regression gate in our testing guide).
- Control cost: run on pull requests that touch prompts, tools or model config, and run everything nightly.
Prefer a managed runner? LangSmith’s pytest plugin (@pytest.mark.langsmith), Langfuse’s langfuse/experiment-action, Braintrust’s braintrustdata/eval-action and Opik’s llm_unit decorator all record each CI run as an experiment, which gives your team a pass/fail trend across runs that a CI log does not.
5. Online evals, canaries and automatic rollback
CI only covers failures you already know about, so watch each release on live traffic before everyone gets it:
- Tag every trace with its release.
- Route a slice of traffic, say 10%, to the new version through a feature flag or your model-routing config.
- Score both versions with the same online evaluators over the same window, so a busy Monday is not blamed on the release.
- Decide automatically: promote, wait or roll back.
- Send the traces that caused a rollback to step 2. They are your next regression cases.
There are three ways to wire the rollback.
A. A flag platform with guarded rollouts. LaunchDarkly’s guarded rollouts raise traffic step by step while monitoring metrics and, if you enable it, roll back automatically when a regression is detected. Its AgentControl configs (previously called AI Configs) can attach built-in accuracy, relevance and toxicity judges that score live responses from 0.0 to 1.0 (in the UI, for completion-mode configs), and its docs say a guarded rollout can be paused or reverted when those scores fall below a threshold.
B. A canary controller. If the agent runs on Kubernetes, export your judge pass rate as a Prometheus metric labelled by version. An Argo Rollouts AnalysisTemplate can query it with a condition such as result[0] >= 0.85 and a failureLimit; when the analysis fails, the rollout aborts and the canary weight goes back to zero.
C. Alert, then switch the config. Every eval platform in the comparison can alert on a score drop. Point the alert’s webhook at a job that sets the canary weight to zero; Langfuse can even trigger a GitHub Actions workflow_dispatch directly. If you already route requests between models, that routing config is the natural switch (see our guide to auto model routing).
Or decide in code. This check takes judged pass and total counts for both versions from the same window, waits for enough data, and rolls back if the canary falls below a floor or is worse than stable by more than noise would explain:
# canary_guard.py: run every 5 minutes from cron, a CI schedule or a Kubernetes CronJob
import json
from math import sqrt
from pathlib import Path
ROUTING = Path("config/routing.json") # read by the app: {"stable": "agent-v41", "canary": "agent-v42", "canary_weight": 0.1}
def decide(stable_pass, stable_n, canary_pass, canary_n,
min_n=200, floor=0.85, max_drop=0.03, z_crit=1.645):
"""Compare judge pass rates for stable vs canary over the same time window."""
if min(stable_n, canary_n) < min_n:
return "wait" # not enough judged traces yet
p_stable, p_canary = stable_pass / stable_n, canary_pass / canary_n
if p_canary < floor:
return "rollback" # below the absolute quality floor
pooled = (stable_pass + canary_pass) / (stable_n + canary_n)
se = sqrt(pooled * (1 - pooled) * (1 / stable_n + 1 / canary_n))
z = (p_stable - p_canary) / se if se else 0.0
if p_stable - p_canary > max_drop and z > z_crit:
return "rollback" # worse than stable, and unlikely to be noise
return "ok"
def set_canary_weight(weight):
config = json.loads(ROUTING.read_text())
config["canary_weight"] = weight
ROUTING.write_text(json.dumps(config, indent=2))
if __name__ == "__main__":
# Judged pass/total counts per version for the last 30 minutes, exported from
# your eval platform's API or your metrics store by an earlier pipeline step.
counts = json.loads(Path("canary_counts.json").read_text())
verdict = decide(**counts)
print(verdict, counts)
if verdict == "rollback":
set_canary_weight(0.0)With 500 judged traces each, stable at 93% and canary at 88% returns rollback (a 5-point drop, z ≈ 2.7); stable at 92% and canary at 90.4% returns ok. Three cautions:
- Rollback must be a config change, not a redeploy, or it will not happen at 2am.
- Frequent checks mean many looks at the same data and more false alarms. Keep thresholds conservative or require two bad checks in a row; LaunchDarkly’s guarded rollouts use sequential testing for this reason.
- Do not change the judge mid-rollout; scores would move for reasons unrelated to the release.
Platform comparison
Features as described in each vendor’s documentation, checked on November 20, 2026. “Not found” means we did not find it documented, not that it cannot be built.
| Platform | OpenTelemetry ingest | Trace to dataset | Online evals on live traffic | CI gate | Score alerts | Rolls back a release |
|---|---|---|---|---|---|---|
| LangSmith | Yes, maps gen_ai.* attributes | Automation rules add filtered, sampled traces to a dataset or annotation queue | LLM-as-judge and code evaluators, with sampling and filters | pytest plugin; each run becomes an experiment | Feedback score, errors, latency, cost; Slack, PagerDuty, Dynatrace, webhook | No |
| Langfuse | Yes; the SDK is built on OpenTelemetry | “+ Add to dataset” on any observation, in bulk, or via SDK; versioned datasets | LLM-as-a-judge on live observations via rules; code evaluators | langfuse/experiment-action fails on RegressionError and comments on the PR | Score, cost, latency; Slack, PagerDuty, webhook, GitHub Actions | No (an alert can dispatch a rollback workflow) |
| Braintrust | Yes, OTLP trace endpoint | “+ Dataset” on selected logs, or insert via SDK or API | Online scoring rules at trace, span or group scope (for example, a session) | braintrustdata/eval-action posts results on PRs | Time-window alerts on SQL aggregates such as average score; Slack, webhook | No |
| Arize Phoenix / AX | Yes (OpenInference on OpenTelemetry) | Add spans to a dataset in the UI; client libraries keep span links | Continuous online eval tasks documented for Arize AX, not open-source Phoenix | Experiments via client or CLI; no official action found | AX monitors on eval labels; Slack, Teams, PagerDuty, OpsGenie | No |
| W&B Weave | Yes, OTLP trace endpoint | “Add selected rows to a dataset” from Traces, or Dataset.from_calls | Signals (0–1 ratings and tags) on agent turns with trend charts; guardrails can block responses inline | Evaluations via SDK; no official action found | Not found | No |
| Opik | Yes (HTTP only) | “Add to dataset” from the traces table, or REST from trace IDs | Online evaluation rules: LLM judge, decision model or Python code, with sampling | pytest llm_unit decorator logs an experiment | Feedback score, errors, cost, latency; Slack, PagerDuty, webhook | No |
| MLflow | Yes, ingests and exports GenAI conventions | search_traces + merge_records into an evaluation dataset | Registered judges started with a sample rate and filter | Run evaluations from your own CI job | Not found | No (registry rollback is metadata; traffic moves at your load balancer) |
And the two release tools that do roll back on their own:
| Tool | What triggers rollback | Quality signal |
|---|---|---|
| LaunchDarkly guarded rollouts | A statistically significant regression on a monitored metric, with automatic rollback enabled | Built-in or custom judges on AgentControl configs |
| Argo Rollouts | An analysis run that exceeds its failureLimit | Any Prometheus query, such as your exported judge pass rate |
How to choose: on LangChain or LangGraph, LangSmith is the shortest path from trace to dataset. For self-hosting or open source, Langfuse covers the whole loop, including a CI action and alerts that can start a GitHub workflow. If your team already runs Arize, W&B, Comet or Databricks, start with what you have. Pick the rollback layer separately.
Where to start this week
- Add a release tag and full tool-call capture to your traces.
- Create one rule that routes negative feedback and errors to a review queue.
- Take the five worst confirmed failures and write them as JSONL cases with fixtures.
- Add the pytest runner to CI for pull requests that touch prompts, tools or model config.
- Ship the next change to 10% of traffic behind a flag, with one online judge and a rollback path you have tested once.
If you want this set up for you, our AI agent development team builds agents with tracing, eval gates and rollback from the first release, and our agentic AI consulting team can add the loop to agents you already run.
FAQ
How do you build a regression test dataset for an AI agent from production failures?
Trace every run and flag failures from user feedback, online evaluator scores and errors. Turn each confirmed failure into a dataset case with the original input, recorded tool responses, the expected behavior and a grader. Check that it fails on the current version, fix the agent, then run the dataset in CI so the bug stays fixed.
Which GenAIOps platforms include automated rollback when an agent’s quality score drops below a threshold?
We found no documented automatic rollback of a release in LangSmith, Langfuse, Braintrust, Arize, W&B Weave, Opik or MLflow; they score live traffic and send alerts. Rollback comes from the release layer: LaunchDarkly guarded rollouts can roll back automatically on a metric regression, Argo Rollouts aborts a canary when its analysis fails, or an alert webhook can switch your feature flag or model-routing config back to the stable version.
Which LLMOps platform converts production failures into regression tests and shows pass/fail trends?
Most cover the core loop. LangSmith automation rules add filtered traces to a dataset automatically, and Langfuse, Braintrust, Opik, Arize Phoenix and W&B Weave add traces to datasets from the UI or SDK. Each shows scores and experiment results in its UI, and most chart them over time. Langfuse and Braintrust have GitHub Actions for pull requests; LangSmith and Opik have pytest integrations.
How do I turn on LangChain tracing?
For LangSmith, set LANGSMITH_TRACING=true and LANGSMITH_API_KEY, plus LANGSMITH_PROJECT in project-based workspaces; no code changes are needed. For Langfuse, pass its CallbackHandler in the callbacks config. For Arize Phoenix, install openinference-instrumentation-langchain and register an OpenTelemetry tracer provider.
Should regression test graders be LLM judges or deterministic checks?
Use deterministic checks wherever possible: which tools were called, their arguments, the output format and forbidden phrases. Use an LLM judge with a written rubric only for semantic failures such as unsupported claims or tone, and compare it with human labels before it can block a merge.
Sources
Vendor documentation as published on November 20, 2026. Products in this space change monthly; check the linked page before relying on a feature.
- OpenTelemetry: GenAI semantic conventions repository
- OpenTelemetry: GenAI agent spans (status: Development)
- OpenTelemetry: GenAI spans, opt-in message content
- LangSmith: Trace LangChain applications
- LangSmith: Trace with OpenTelemetry
- LangSmith: Automation rules
- LangSmith: Online evaluations
- LangSmith: pytest integration
- LangSmith: Alerts
- LangSmith: Dashboards
- Langfuse: OpenTelemetry integration
- Langfuse: LangChain and LangGraph integration
- Langfuse: Evaluation concepts
- Langfuse: Datasets
- Langfuse: Experiments in CI/CD
- Langfuse: Alerts
- Langfuse: Score analytics
- Braintrust: OpenTelemetry
- Braintrust: Score online
- Braintrust: Alerts
- Braintrust: Run in CI
- Braintrust: Turn LLM production failures into regression tests
- Arize Phoenix: Set up tracing with phoenix.otel
- Arize Phoenix: Create datasets from traces (release note)
- Arize AX: Run evals on traces
- Arize AX: Production monitoring
- W&B Weave: OpenTelemetry
- W&B Weave: Datasets
- W&B Weave: Agent signals
- W&B Weave: Guardrails
- Opik: OpenTelemetry
- Opik: Manage datasets
- Opik: Online evaluation rules
- Opik: Alerts
- Opik: llm_unit reference
- MLflow: OpenTelemetry
- MLflow: Production tracing and online evaluation
- MLflow: Evaluation datasets
- MLflow: Canary deployment for AI models
- LaunchDarkly: Guarded rollouts
- LaunchDarkly: Online evaluations
- LaunchDarkly: AgentControl API (formerly AI Configs)
- Argo Rollouts: Analysis and progressive delivery
