Monitoring a deployed system for drift: Asymptotic Prediction-Powered Risk Monitoring¶
AsymptoticPPIMeanMonitor and its label-only counterpart AsymptoticClassicalMeanMonitor watch a quality metric across successive batches of production data and raise an alarm the moment there is statistically valid evidence that it has drifted past a threshold you fix in advance. This is necessary because a deployed system's quality is rarely static: traffic shifts and upstream models change, so a metric that looked fine at launch can drift silently, and a single-shot estimate made at first deployment time cannot catch that.
What you will learn:
- Why re-checking a fresh confidence interval multiple times inflates the false-alarm rate
- How
AsymptoticPPIMeanMonitorandAsymptoticClassicalMeanMonitorgive an anytime-valid alternative that stays safe under repeated checks - How adding proxy labels lets
AsymptoticPPIMeanMonitordetect a drift sooner than the label-onlyAsymptoticClassicalMeanMonitor - How to read their running-mean curves, confidence bounds, and alarms
The problem: the metric can drift after launch, and repeated checking breaks validity¶
Suppose you have a customer-facing AI assistant that handles an open-ended troubleshooting flow, and hallucinations are a real risk there. Every week, a fresh batch of conversations is judged by your LLM judge, and a handful are sent for human review. Your team has fixed a business threshold on the hallucination rate, and wants to know as soon as the rate has drifted past it.
The tempting approach is to compute a fresh confidence interval every week and raise an alarm the moment one crosses the threshold. This is peeking: each weekly test carries its own false-alarm probability, and checking after every batch compounds those chances, so a false alarm becomes likely over a long enough monitoring horizon even if nothing has actually drifted. The Monitors user guide explains why this happens and how an anytime-valid confidence sequence that stays safe no matter how often you look avoids it. This tutorial focuses on what that means for you in practice.
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from glide.estimators import PPIMeanEstimator
from glide.monitors import AsymptoticClassicalMeanMonitor, AsymptoticPPIMeanMonitor
from glide.samplers import UniformSampler
from glide.simulators import generate_stratified_binary_dataset, simulate_annotation
# ── Colour palette ──────────────────────────────────────────
C_ESTIMATE = "#95A5A6" # per-batch point estimate — grey
C_NAIVE = "#E74C3C" # naive baseline alarm — red-orange
C_PPI = "#27AE60" # PPI monitor running mean/bound — green
C_CLASSICAL = "#2980B9" # classical monitor running mean/bound — blue
C_ALARM_PPI = "#8E44AD" # PPI monitor alarm — purple
C_ALARM_CLASSICAL = "#F39C12" # classical monitor alarm — gold
C_THRESHOLD = "#2C3E50" # threshold — dark slate
# ── Global plot style ─────────────────────────────────
plt.rcParams.update(
{
"figure.facecolor": "white",
"axes.facecolor": "#FAFAFA",
"axes.grid": True,
"grid.color": "#E5E5E5",
"grid.linewidth": 0.8,
"font.size": 18,
"axes.labelsize": 18,
"axes.titlesize": 18,
"legend.fontsize": 14,
"xtick.labelsize": 16,
"ytick.labelsize": 16,
"figure.titlesize": 19,
}
)
Simulating a batched production stream¶
We use generate_stratified_binary_dataset to simulate successive batches of conversations, each with its own true hallucination rate, proxy rate, and correlation, stacked into a single stream with an integer identifying the chronological batch each row belongs to.
simulate_batches below builds such a stream from a list of per-week true hallucination rates. Every week gets BATCH_SIZE proxy-judged conversations, of which LABELS_PER_BATCH are sent for human review: we use the UniformSampler to simulate uniform allocation of the review budget at each week. The human labels are revealed only for selected conversations, leaving the rest as np.nan. We fix an alarm threshold above the stationary rate and assess the monitoring methods' ability to detect when it is exceeded while avoiding false alarms. All batches in the stream are monitored.
N_BATCHES = 50
BATCH_SIZE = 800
LABELS_PER_BATCH = 40
STATIONARY_MEAN = 0.28
PROXY_BIAS = 0.05
CORRELATION = 0.85
CONFIDENCE_LEVEL = 0.8
THRESHOLD = 0.3
def simulate_batches(true_means, random_seed):
proxy_means = true_means + PROXY_BIAS
correlations = np.full(len(true_means), CORRELATION)
y_true_oracle, y_proxy, batches = generate_stratified_binary_dataset(
n_total=np.full(len(true_means), BATCH_SIZE),
true_mean=true_means,
proxy_mean=proxy_means,
correlation=correlations,
random_seed=random_seed,
)
xi = np.hstack(
[UniformSampler().sample(BATCH_SIZE, LABELS_PER_BATCH, random_seed=random_seed + i) for i in range(N_BATCHES)]
)
y_true = simulate_annotation(y_true_oracle, xi)
return y_true, y_proxy, batches
The naive baseline: one confidence interval per batch¶
The tempting-but-invalid approach simply calls PPIMeanEstimator once per week, comparing that week's confidence interval to the threshold and alarming whenever the interval lies above it.
def naive_per_batch_alarms(y_true, y_proxy, batches, threshold, n_batches):
means = np.zeros(n_batches)
lower_bounds = np.zeros(n_batches)
alarms = np.zeros(n_batches, dtype=bool)
for batch_id in range(n_batches):
batch_mask = batches == batch_id
batch_result = PPIMeanEstimator().estimate(y_true[batch_mask], y_proxy[batch_mask])
means[batch_id] = batch_result.mean
lower_bounds[batch_id] = batch_result.confidence_interval.lower_bound
alarms[batch_id] = batch_result.confidence_interval.lower_bound > threshold
return means, lower_bounds, alarms
def plot_monitor_vs_baseline(batch_means, ppi_result, classical_result, naive_alarms, title):
weeks = np.arange(1, len(batch_means) + 1)
fig, ax = plt.subplots(figsize=(10, 6.8))
ax.scatter(weeks, batch_means, color=C_ESTIMATE, s=28, alpha=0.8, zorder=3, label="Per-batch estimate")
ax.plot(weeks, ppi_result.running_means, color=C_PPI, linewidth=2.5, zorder=4, label="PPI monitor running mean")
ax.plot(
weeks,
ppi_result.confidence_bounds,
color=C_PPI,
linewidth=2.5,
linestyle="--",
zorder=4,
label="PPI monitor lower bound",
)
ax.plot(
weeks,
classical_result.running_means,
color=C_CLASSICAL,
linewidth=2.5,
zorder=4,
label="Classical monitor running mean",
)
ax.plot(
weeks,
classical_result.confidence_bounds,
color=C_CLASSICAL,
linewidth=2.5,
linestyle="--",
zorder=4,
label="Classical monitor lower bound",
)
ax.axhline(THRESHOLD, color=C_THRESHOLD, linestyle="--", linewidth=1.8, zorder=2, label="Threshold")
if naive_alarms.any():
ax.scatter(
weeks[naive_alarms],
batch_means[naive_alarms],
marker="x",
s=110,
linewidths=2.5,
color=C_NAIVE,
zorder=5,
label="Naive baseline alarm",
)
if ppi_result.drift_detected:
alarm_week = ppi_result.first_alarm_index + 1
ax.axvline(
alarm_week,
color=C_ALARM_PPI,
linestyle=":",
linewidth=2.2,
zorder=2,
label=f"PPI alarm (week {alarm_week})",
)
if classical_result.drift_detected:
alarm_week = classical_result.first_alarm_index + 1
ax.axvline(
alarm_week,
color=C_ALARM_CLASSICAL,
linestyle=":",
linewidth=2.2,
zorder=2,
label=f"Classical alarm (week {alarm_week})",
)
ax.set_xlabel("Week")
ax.set_ylabel("Hallucination Rate")
ax.yaxis.set_major_formatter(plt.FuncFormatter(lambda v, _: f"{v:.0%}"))
ax.set_title(title)
ax.set_ylim(-0.05, 1.0)
ax.spines[["top", "right"]].set_visible(False)
ax.legend(loc="upper left", framealpha=0.9)
plt.tight_layout()
plt.show()
Scenario 1: no drift, no false alarm¶
We generate a fully stationary stream: the true hallucination rate stays below the threshold. Neither method should alarm, except by chance.
true_means_no_drift = np.full(N_BATCHES, STATIONARY_MEAN)
y_true_no_drift, y_proxy_no_drift, batches_no_drift = simulate_batches(true_means_no_drift, random_seed=7)
n_labeled = np.sum(~np.isnan(y_true_no_drift))
print(f"Total conversations : {len(y_true_no_drift):,}")
print(f" Human-reviewed : {n_labeled:,}")
print(f" Number of weeks : {len(np.unique(batches_no_drift))}")
Total conversations : 40,000 Human-reviewed : 2,000 Number of weeks : 50
Run the naive baseline and both monitoring methods.
means_no_drift, lower_no_drift, naive_alarms_no_drift = naive_per_batch_alarms(
y_true_no_drift, y_proxy_no_drift, batches_no_drift, THRESHOLD, N_BATCHES
)
ppi_result_no_drift = AsymptoticPPIMeanMonitor().detect(
y_true_no_drift,
y_proxy_no_drift,
batches_no_drift,
higher_is_better=False,
threshold=THRESHOLD,
confidence_level=CONFIDENCE_LEVEL,
metric_name="Hallucination Rate",
)
classical_result_no_drift = AsymptoticClassicalMeanMonitor().detect(
y_true_no_drift,
batches_no_drift,
higher_is_better=False,
threshold=THRESHOLD,
confidence_level=CONFIDENCE_LEVEL,
metric_name="Hallucination Rate",
)
naive_alarm_weeks = np.arange(1, N_BATCHES + 1)[naive_alarms_no_drift]
print(f"Naive baseline alarms at week(s): {naive_alarm_weeks}")
print(f"PPI monitor drift detected: {ppi_result_no_drift.drift_detected}")
print(f"Classical monitor drift detected: {classical_result_no_drift.drift_detected}")
print(
f"PPI monitor running mean / bound after {N_BATCHES} weeks: {ppi_result_no_drift.running_means[-1]:.1%} / "
f"{ppi_result_no_drift.confidence_bounds[-1]:.1%}"
)
print(
f"Classical monitor running mean / bound after {N_BATCHES} weeks: "
f"{classical_result_no_drift.running_means[-1]:.1%} / {classical_result_no_drift.confidence_bounds[-1]:.1%}"
)
Naive baseline alarms at week(s): [ 4 37] PPI monitor drift detected: False Classical monitor drift detected: False PPI monitor running mean / bound after 50 weeks: 27.9% / 26.6% Classical monitor running mean / bound after 50 weeks: 28.3% / 26.1%
plot_monitor_vs_baseline(
means_no_drift,
ppi_result_no_drift,
classical_result_no_drift,
naive_alarms_no_drift,
title="Scenario 1: stationary stream (no drift)",
)
At some point, ordinary sampling noise pushes one of the confidence intervals above the threshold, and the naive baseline raises false alarms, purely from peeking: with a fresh, independent-looking test every week, the chance of at least one false alarm climbs the longer you keep checking, even though the hallucination rate did not move. AsymptoticPPIMeanMonitor and AsymptoticClassicalMeanMonitor, in contrast, never alarm across the full stream: their running means stay below the threshold, with anytime-valid lower bounds safely underneath it as well.
Scenario 2: slow drift, early alarm¶
Now we generate data with a progressive drift: partway through the stream, the hallucination rate ramps up to an unacceptably high level.
DRIFT_START = 15
DRIFT_AMPLITUDE = 0.35
RAMP_LENGTH = 30
true_means_slow_drift = np.hstack(
[
np.full(DRIFT_START, STATIONARY_MEAN),
np.linspace(STATIONARY_MEAN, STATIONARY_MEAN + DRIFT_AMPLITUDE, RAMP_LENGTH),
np.full(N_BATCHES - RAMP_LENGTH - DRIFT_START, STATIONARY_MEAN + DRIFT_AMPLITUDE),
]
)
y_true_slow_drift, y_proxy_slow_drift, batches_slow_drift = simulate_batches(true_means_slow_drift, random_seed=0)
means_slow_drift, lower_slow_drift, naive_alarms_slow_drift = naive_per_batch_alarms(
y_true_slow_drift, y_proxy_slow_drift, batches_slow_drift, THRESHOLD, N_BATCHES
)
ppi_result_slow_drift = AsymptoticPPIMeanMonitor().detect(
y_true_slow_drift,
y_proxy_slow_drift,
batches_slow_drift,
higher_is_better=False,
threshold=THRESHOLD,
confidence_level=CONFIDENCE_LEVEL,
metric_name="Hallucination Rate",
)
classical_result_slow_drift = AsymptoticClassicalMeanMonitor().detect(
y_true_slow_drift,
batches_slow_drift,
higher_is_better=False,
threshold=THRESHOLD,
confidence_level=CONFIDENCE_LEVEL,
metric_name="Hallucination Rate",
)
naive_alarm_weeks = np.flatnonzero(naive_alarms_slow_drift) + 1
print(f"Naive baseline alarms: {len(naive_alarm_weeks)} weeks, first at week {naive_alarm_weeks[0]}")
ppi_alarm_index = ppi_result_slow_drift.first_alarm_index
if ppi_alarm_index is not None:
print(f"PPI monitor drift detected: True, first alarm at week {ppi_alarm_index + 1}")
classical_alarm_index = classical_result_slow_drift.first_alarm_index
if classical_alarm_index is not None:
print(f"Classical monitor drift detected: True, first alarm at week {classical_alarm_index + 1}")
Naive baseline alarms: 27 weeks, first at week 18 PPI monitor drift detected: True, first alarm at week 25 Classical monitor drift detected: True, first alarm at week 32
plot_monitor_vs_baseline(
means_slow_drift,
ppi_result_slow_drift,
classical_result_slow_drift,
naive_alarms_slow_drift,
title="Scenario 2: drift occurrence during deployment",
)
Observe that the peeking baseline alarms earlier. However, this occurs even before the drift happens and, therefore, qualifies as a false alarm. In real time, a team watching the naive baseline has no way to tell a validated alarm from a lucky (or unlucky) one, because its error rate was never controlled for repeated testing in the first place: Scenario 1 already showed it firing on a completely stationary stream. On the other hand, the anytime-valid alarms come fairly early after a consistent drift can be observed and not before, which makes them more trustworthy. Between the two, AsymptoticPPIMeanMonitor alarms sooner than AsymptoticClassicalMeanMonitor: the correlated proxy labels sharpen its per-batch estimate on top of the human labels both monitors share, shrinking its confidence sequence and letting it accumulate evidence of the drift faster.
AsymptoticPPIMeanMonitor and AsymptoticClassicalMeanMonitor's alarms carry no ambiguity: each is backed by the same anytime-valid guarantee demonstrated in Scenario 1, so under no drift the chance either would ever fire at all across the whole monitoring horizon corresponds to the used confidence level. At the moment each fires, its anytime-valid bound has just crossed the threshold. By the end of the stream, both running means and their bounds are clearly above the threshold.
Reading the result object¶
In both scenarios, AsymptoticPPIMeanMonitor returned a PredictionPoweredMeanMonitoringResult. Its public surface covers everything used above, plus a formatted summary.
print("PPI monitor")
print(f"{'Drift detected':<32}: {ppi_result_slow_drift.drift_detected}")
print(f"{'First alarm index':<32}: {ppi_result_slow_drift.first_alarm_index}")
print(f"{'Alarm threshold':<32}: {ppi_result_slow_drift.alarm_threshold}")
print(f"{'Confidence level':<32}: {ppi_result_slow_drift.confidence_level}")
print(f"{'Running means (last 3)':<32}: {np.round(ppi_result_slow_drift.running_means[-3:], 3)}")
print(f"{'Confidence bounds (last 3)':<32}: {np.round(ppi_result_slow_drift.confidence_bounds[-3:], 3)}")
print(f"{'True labels per batch (last 3)':<32}: {ppi_result_slow_drift.batch_n_true[-3:]}")
print(f"{'Proxy labels per batch (last 3)':<32}: {ppi_result_slow_drift.batch_n_proxy[-3:]}")
PPI monitor Drift detected : True First alarm index : 24 Alarm threshold : 0.3 Confidence level : 0.8 Running means (last 3) : [0.429 0.433 0.438] Confidence bounds (last 3) : [0.417 0.421 0.425] True labels per batch (last 3) : [40 40 40] Proxy labels per batch (last 3) : [800 800 800]
Summary : Anytime-valid Risk Monitoring detects drifts and reduces false alarms¶
| Per-batch baseline | AsymptoticClassicalMeanMonitor |
AsymptoticPPIMeanMonitor |
|
|---|---|---|---|
| Valid under repeated looks? | ❌ No | ✅ Yes | ✅ Yes |
| Uses proxy labels? | ✅ Yes | ❌ No | ✅ Yes |
| Detects a sustained drift? | 🟠 Yes, but prone to early false alarms | ✅ Yes | ✅ Yes, sooner than the classical monitor |
Key takeaways:
- Repeated significance testing is not a monitor. Scenario 1 showed the per-batch baseline raise a false alarm on a perfectly stationary stream, purely from checking every week.
- An anytime-valid confidence sequence makes checking every week safe. Both
AsymptoticPPIMeanMonitorandAsymptoticClassicalMeanMonitor's guarantees hold simultaneously across all looks, so their alarms in Scenario 2 can be trusted the moment they fire, without waiting to see whether later weeks confirm them. - Adding proxy labels speeds up detection.
AsymptoticPPIMeanMonitoralarmed sooner thanAsymptoticClassicalMeanMonitorin Scenario 2, since its confidence sequence is sharpened by the correlated proxy labels on top of the human labels both monitors share. - The naive baseline's alarms carry no error-rate guarantee, correct or not. Scenario 2's naive alarms happened to line up with a real regression, but Scenario 1 already showed the same procedure crying wolf under no drift at all: there is no way to tell the two situations apart from the naive alarms alone.
Want to go further? The Asymptotic PPRM scientific validation notebook provides rigorous empirical evidence on false-alarm control and detection power, the Monitors user guide derives the anytime-valid confidence sequence used here from first principles.