How to Build Regression Test Datasets for AI Agents from Production Failures

By stackcone · November 2026

To build regression tests from production failures, run one loop continuously: trace every agent run, flag the runs that went wrong (user feedback, online evaluator scores, errors), turn each confirmed failure into a dataset case with the original input, recorded tool responses and a grader, run that dataset in CI as a merge gate, and watch each release on a canary with online evals, rolling back when the score drops.

The tooling answer is just as short. LangSmith, Langfuse, Braintrust, Arize Phoenix, W&B Weave, Opik and MLflow all ingest traces and can turn a trace into a dataset item, and most of them score live traffic and alert when a score falls. We found no documented automatic rollback of a release in any of them. That job belongs to the release layer: a feature flag with guarded rollouts, a canary controller such as Argo Rollouts, or a small script that switches your model-routing config back to the stable version. All three are covered below, with working code.

Our AI agent testing and evaluation guide covers golden datasets and trace replay in general; this post goes deep on the loop from a production failure to a test that keeps it fixed.

The loop at a glance

StepWhat you doWhat you end up with
1. CaptureTrace every run: input, versions, tool calls and results, outputA trace you can replay
2. TriageRoute bad feedback, low judge scores and errors to review; label the failure modeConfirmed, labelled failures
3. PromoteCopy the failure into a dataset with tool fixtures, expected behavior and a graderA regression case linked to its trace
4. GateRun the dataset on every pull requestA merge that cannot bring the bug back
5. WatchCanary the new version, score it live, roll back if it is worseBad releases that undo themselves, and new failures for step 2

1. Capture traces you can replay

A failure you cannot reproduce cannot become a test. A useful trace holds everything needed to rerun the turn without touching production systems:

Use OpenTelemetry where you can. The OpenTelemetry GenAI semantic conventions now live in their own repository and are still marked Development, so names can change. They define agent operations such as invoke_agent and execute_tool, and attributes such as gen_ai.operation.name, gen_ai.agent.name, gen_ai.conversation.id, gen_ai.request.model and gen_ai.usage.input_tokens. Message content such as gen_ai.output.messages is opt-in because it can hold personal data, so decide deliberately what you record (our GDPR guide for AI agents covers retention). All seven platforms compared below accept OTLP traces, so instrumenting once keeps you free to switch.

LangChain tracing in practice

DestinationHow LangChain and LangGraph runs get there
LangSmithSet LANGSMITH_TRACING=true and LANGSMITH_API_KEY; add LANGSMITH_PROJECT in project-based workspaces. No code changes. LANGSMITH_OTEL_ENABLED=true switches export to OpenTelemetry.
Langfusefrom langfuse.langchain import CallbackHandler, then pass the handler in config={"callbacks": [handler]} when you call invoke.
Arize PhoenixInstall openinference-instrumentation-langchain and call register() from phoenix.otel with auto-instrumentation, or use LangChainInstrumentor.

In serverless functions, flush before exit (LangSmith’s docs suggest synchronous callbacks or wait_for_all_tracers() in Python), or you lose the traces of timeouts and crashes, the failures you most need.

2. Triage: find the failures worth keeping

Most traces are fine. Combine several signals to surface the ones that are not; each signal misses different failures:

SignalHow to capture itWhat it misses
Explicit feedbackThumbs down or a rating, logged as a score on the traceMost users never click
Implicit feedbackUser rephrases the request, asks for a human, abandons the task, or rewrites the agent’s draftNeeds product events joined to traces
Online evaluatorsLLM judge or code check on a sample of live traces (groundedness, policy, task completed)Judge errors; cost limits sampling
Error signalsTool exceptions, error.type on spans, timeouts, step-limit hits, schema failuresConfident wrong answers raise no error
OutliersRuns with far more steps, tokens or cost than the median for that taskShort wrong answers

Automate the routing, not the judgment. A LangSmith automation rule, for example, takes a filter (negative feedback, a low evaluator score, an error), a sampling rate and an action such as add to annotation queue or add to dataset; the other platforms offer similar rules or review queues. Someone then reviews the queue on a schedule, confirms each failure and tags its mode:

Counting failures by mode shows where to spend effort, and near-identical failures become one case, not twenty.

3. Turn a failure into a dataset case

Say a customer wrote “I was charged twice for order 8812, please refund one” and the agent cancelled the order instead. Here is that failure as a regression case. In the file it is one JSON object per line (JSONL); it is spread out here for reading:

{
  "id": "dup-charge-refund-8812",
  "source_trace": "trace_7f3a9c",
  "failure_mode": "wrong_tool",
  "input": {"messages": [{"role": "user", "content": "I was charged twice for order 8812, please refund one"}]},
  "fixtures": {"get_order": {"id": "8812", "charges": [49.0, 49.0]}},
  "expect": {
    "tools_called": ["get_order", "create_refund"],
    "tools_not_called": ["cancel_order"],
    "args": {"create_refund": {"amount": 49.0}},
    "output_contains": ["refund"]
  },
  "trials": 3,
  "min_pass": 3
}
  1. Reproduce first. Run the case on the current version and confirm it fails. If it passes, you missed some context or the grader is too loose.
  2. Freeze the world with fixtures. Copy the tool responses from the trace into fixtures and have your harness return them instead of calling live APIs, so the test checks the agent’s decisions, not today’s database.
  3. Write the expectation as behavior. “Calls create_refund for 49.0 and never cancel_order” survives a prompt rewrite; a golden output string does not.
  4. Pick the cheapest grader that catches the failure (table below).
  5. Keep provenance: source trace, failure mode, date and the version that failed.
  6. Strip personal data before the case lands where developers and CI can read it.
Failure modeGrader
wrong_tool, missing_action, loopCheck the tool-call sequence and count
bad_argsCompare arguments to expected values
formatJSON schema or regex validation
retrieval_missExpected document ID in the top k results
unsupported_claim, tone, policy wordingLLM judge with a written rubric, calibrated on human labels

Deterministic graders are free, fast and consistent. Use an LLM judge only where meaning matters, give it a rubric with pass and fail examples, and check it against a few dozen human labels before it can block a merge (Langfuse’s score analytics, for one, shows judge-versus-human agreement). Every platform in the comparison below can create the dataset item from a trace for you; you still add the expectation and the grader.

4. Run the dataset in CI as a regression gate

A golden set gates on an overall pass rate. A case from a real incident should be must-pass: the bug was fixed, so any failure means it is back. Agents are not deterministic, so each case runs several times and has its own minimum.

This pytest runner runs each case trials times with its fixtures, applies the checks in expect, and fails the build if a case passes fewer than min_pass times. Your run_agent must return the final output and the list of tool calls.

# evals/test_regressions.py
import json
from pathlib import Path

import pytest

from my_agent import run_agent  # returns {"output": str, "tool_calls": [{"name": str, "args": dict}]}

DATASET = Path(__file__).with_name("regressions.jsonl")
CASES = [json.loads(line) for line in DATASET.read_text().splitlines() if line.strip()]


def grade(case, result):
    """Return a list of failure reasons. An empty list means the run passed."""
    expect, errors = case["expect"], []
    called = [c["name"] for c in result["tool_calls"]]

    remaining = iter(called)  # expected tools must appear in this order; extra calls are allowed
    if not all(name in remaining for name in expect.get("tools_called", [])):
        errors.append(f"expected tools {expect['tools_called']} in order, got {called}")
    for name in expect.get("tools_not_called", []):
        if name in called:
            errors.append(f"called forbidden tool {name}")
    for name, want in expect.get("args", {}).items():
        got = next((c["args"] for c in result["tool_calls"] if c["name"] == name), {})
        for key, value in want.items():
            if got.get(key) != value:
                errors.append(f"{name}.{key}: expected {value!r}, got {got.get(key)!r}")

    text = result["output"].lower()
    errors += [f"output missing {s!r}" for s in expect.get("output_contains", []) if s.lower() not in text]
    errors += [f"output contains {s!r}" for s in expect.get("output_not_contains", []) if s.lower() in text]
    return errors


@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_production_regression(case):
    trials, min_pass = case.get("trials", 1), case.get("min_pass", 1)
    failures = []
    for _ in range(trials):
        result = run_agent(case["input"], fixtures=case.get("fixtures", {}))
        errors = grade(case, result)
        if errors:
            failures.append(errors)
    passed = trials - len(failures)
    assert passed >= min_pass, f"passed {passed}/{trials}, first failure: {failures[0]}"

Run it with python -m pytest evals/ from the project root. A failure reads like passed 0/3, first failure: ['called forbidden tool cancel_order'] under the case ID, so the CI log points straight at the incident.

Prefer a managed runner? LangSmith’s pytest plugin (@pytest.mark.langsmith), Langfuse’s langfuse/experiment-action, Braintrust’s braintrustdata/eval-action and Opik’s llm_unit decorator all record each CI run as an experiment, which gives your team a pass/fail trend across runs that a CI log does not.

5. Online evals, canaries and automatic rollback

CI only covers failures you already know about, so watch each release on live traffic before everyone gets it:

  1. Tag every trace with its release.
  2. Route a slice of traffic, say 10%, to the new version through a feature flag or your model-routing config.
  3. Score both versions with the same online evaluators over the same window, so a busy Monday is not blamed on the release.
  4. Decide automatically: promote, wait or roll back.
  5. Send the traces that caused a rollback to step 2. They are your next regression cases.

There are three ways to wire the rollback.

A. A flag platform with guarded rollouts. LaunchDarkly’s guarded rollouts raise traffic step by step while monitoring metrics and, if you enable it, roll back automatically when a regression is detected. Its AgentControl configs (previously called AI Configs) can attach built-in accuracy, relevance and toxicity judges that score live responses from 0.0 to 1.0 (in the UI, for completion-mode configs), and its docs say a guarded rollout can be paused or reverted when those scores fall below a threshold.

B. A canary controller. If the agent runs on Kubernetes, export your judge pass rate as a Prometheus metric labelled by version. An Argo Rollouts AnalysisTemplate can query it with a condition such as result[0] >= 0.85 and a failureLimit; when the analysis fails, the rollout aborts and the canary weight goes back to zero.

C. Alert, then switch the config. Every eval platform in the comparison can alert on a score drop. Point the alert’s webhook at a job that sets the canary weight to zero; Langfuse can even trigger a GitHub Actions workflow_dispatch directly. If you already route requests between models, that routing config is the natural switch (see our guide to auto model routing).

Or decide in code. This check takes judged pass and total counts for both versions from the same window, waits for enough data, and rolls back if the canary falls below a floor or is worse than stable by more than noise would explain:

# canary_guard.py: run every 5 minutes from cron, a CI schedule or a Kubernetes CronJob
import json
from math import sqrt
from pathlib import Path

ROUTING = Path("config/routing.json")  # read by the app: {"stable": "agent-v41", "canary": "agent-v42", "canary_weight": 0.1}


def decide(stable_pass, stable_n, canary_pass, canary_n,
           min_n=200, floor=0.85, max_drop=0.03, z_crit=1.645):
    """Compare judge pass rates for stable vs canary over the same time window."""
    if min(stable_n, canary_n) < min_n:
        return "wait"  # not enough judged traces yet
    p_stable, p_canary = stable_pass / stable_n, canary_pass / canary_n
    if p_canary < floor:
        return "rollback"  # below the absolute quality floor
    pooled = (stable_pass + canary_pass) / (stable_n + canary_n)
    se = sqrt(pooled * (1 - pooled) * (1 / stable_n + 1 / canary_n))
    z = (p_stable - p_canary) / se if se else 0.0
    if p_stable - p_canary > max_drop and z > z_crit:
        return "rollback"  # worse than stable, and unlikely to be noise
    return "ok"


def set_canary_weight(weight):
    config = json.loads(ROUTING.read_text())
    config["canary_weight"] = weight
    ROUTING.write_text(json.dumps(config, indent=2))


if __name__ == "__main__":
    # Judged pass/total counts per version for the last 30 minutes, exported from
    # your eval platform's API or your metrics store by an earlier pipeline step.
    counts = json.loads(Path("canary_counts.json").read_text())
    verdict = decide(**counts)
    print(verdict, counts)
    if verdict == "rollback":
        set_canary_weight(0.0)

With 500 judged traces each, stable at 93% and canary at 88% returns rollback (a 5-point drop, z ≈ 2.7); stable at 92% and canary at 90.4% returns ok. Three cautions:

Platform comparison

Features as described in each vendor’s documentation, checked on November 20, 2026. “Not found” means we did not find it documented, not that it cannot be built.

PlatformOpenTelemetry ingestTrace to datasetOnline evals on live trafficCI gateScore alertsRolls back a release
LangSmithYes, maps gen_ai.* attributesAutomation rules add filtered, sampled traces to a dataset or annotation queueLLM-as-judge and code evaluators, with sampling and filterspytest plugin; each run becomes an experimentFeedback score, errors, latency, cost; Slack, PagerDuty, Dynatrace, webhookNo
LangfuseYes; the SDK is built on OpenTelemetry“+ Add to dataset” on any observation, in bulk, or via SDK; versioned datasetsLLM-as-a-judge on live observations via rules; code evaluatorslangfuse/experiment-action fails on RegressionError and comments on the PRScore, cost, latency; Slack, PagerDuty, webhook, GitHub ActionsNo (an alert can dispatch a rollback workflow)
BraintrustYes, OTLP trace endpoint“+ Dataset” on selected logs, or insert via SDK or APIOnline scoring rules at trace, span or group scope (for example, a session)braintrustdata/eval-action posts results on PRsTime-window alerts on SQL aggregates such as average score; Slack, webhookNo
Arize Phoenix / AXYes (OpenInference on OpenTelemetry)Add spans to a dataset in the UI; client libraries keep span linksContinuous online eval tasks documented for Arize AX, not open-source PhoenixExperiments via client or CLI; no official action foundAX monitors on eval labels; Slack, Teams, PagerDuty, OpsGenieNo
W&B WeaveYes, OTLP trace endpoint“Add selected rows to a dataset” from Traces, or Dataset.from_callsSignals (0–1 ratings and tags) on agent turns with trend charts; guardrails can block responses inlineEvaluations via SDK; no official action foundNot foundNo
OpikYes (HTTP only)“Add to dataset” from the traces table, or REST from trace IDsOnline evaluation rules: LLM judge, decision model or Python code, with samplingpytest llm_unit decorator logs an experimentFeedback score, errors, cost, latency; Slack, PagerDuty, webhookNo
MLflowYes, ingests and exports GenAI conventionssearch_traces + merge_records into an evaluation datasetRegistered judges started with a sample rate and filterRun evaluations from your own CI jobNot foundNo (registry rollback is metadata; traffic moves at your load balancer)

And the two release tools that do roll back on their own:

ToolWhat triggers rollbackQuality signal
LaunchDarkly guarded rolloutsA statistically significant regression on a monitored metric, with automatic rollback enabledBuilt-in or custom judges on AgentControl configs
Argo RolloutsAn analysis run that exceeds its failureLimitAny Prometheus query, such as your exported judge pass rate

How to choose: on LangChain or LangGraph, LangSmith is the shortest path from trace to dataset. For self-hosting or open source, Langfuse covers the whole loop, including a CI action and alerts that can start a GitHub workflow. If your team already runs Arize, W&B, Comet or Databricks, start with what you have. Pick the rollback layer separately.

Where to start this week

  1. Add a release tag and full tool-call capture to your traces.
  2. Create one rule that routes negative feedback and errors to a review queue.
  3. Take the five worst confirmed failures and write them as JSONL cases with fixtures.
  4. Add the pytest runner to CI for pull requests that touch prompts, tools or model config.
  5. Ship the next change to 10% of traffic behind a flag, with one online judge and a rollback path you have tested once.

If you want this set up for you, our AI agent development team builds agents with tracing, eval gates and rollback from the first release, and our agentic AI consulting team can add the loop to agents you already run.

FAQ

How do you build a regression test dataset for an AI agent from production failures?

Trace every run and flag failures from user feedback, online evaluator scores and errors. Turn each confirmed failure into a dataset case with the original input, recorded tool responses, the expected behavior and a grader. Check that it fails on the current version, fix the agent, then run the dataset in CI so the bug stays fixed.

Which GenAIOps platforms include automated rollback when an agent’s quality score drops below a threshold?

We found no documented automatic rollback of a release in LangSmith, Langfuse, Braintrust, Arize, W&B Weave, Opik or MLflow; they score live traffic and send alerts. Rollback comes from the release layer: LaunchDarkly guarded rollouts can roll back automatically on a metric regression, Argo Rollouts aborts a canary when its analysis fails, or an alert webhook can switch your feature flag or model-routing config back to the stable version.

Which LLMOps platform converts production failures into regression tests and shows pass/fail trends?

Most cover the core loop. LangSmith automation rules add filtered traces to a dataset automatically, and Langfuse, Braintrust, Opik, Arize Phoenix and W&B Weave add traces to datasets from the UI or SDK. Each shows scores and experiment results in its UI, and most chart them over time. Langfuse and Braintrust have GitHub Actions for pull requests; LangSmith and Opik have pytest integrations.

How do I turn on LangChain tracing?

For LangSmith, set LANGSMITH_TRACING=true and LANGSMITH_API_KEY, plus LANGSMITH_PROJECT in project-based workspaces; no code changes are needed. For Langfuse, pass its CallbackHandler in the callbacks config. For Arize Phoenix, install openinference-instrumentation-langchain and register an OpenTelemetry tracer provider.

Should regression test graders be LLM judges or deterministic checks?

Use deterministic checks wherever possible: which tools were called, their arguments, the output format and forbidden phrases. Use an LLM judge with a written rubric only for semantic failures such as unsupported claims or tone, and compare it with human labels before it can block a merge.

Sources

Vendor documentation as published on November 20, 2026. Products in this space change monthly; check the linked page before relying on a feature.

Agents failing in production?

stackcone sets up agent ops for the agents you run: tracing, regression datasets built from real failures, eval gates in CI, and canary releases that roll back on their own.