Scientific Validity of PPI Mean Monitoring¶
This notebook provides empirical evidence that GLIDE's Prediction-Powered Risk Monitoring (PPRM) implementation is statistically valid.
Setup: Once a GenAI system is deployed, the question shifts from "what is the metric today?" to "has the metric drifted?" GLIDE's monitors re-estimate a metric, treated throughout as a risk (lower is better), on successive batches of production data, and raise an alarm the moment there is statistically valid evidence that it has crossed a threshold. A monitor is judged by:
- False-alarm rate : how often it alarms under no drift
- Detection power and speed : how reliably and how quickly it alarms under actual drift
We compare PPRM and two other methods, each dropping one virtue to isolate its effect.
PPRM (
PPIMeanMonitor) targets the debiased true mean, is anytime-valid, and uses the proxy.Classical (
ClassicalMeanMonitor) also targets a true-risk estimand and is anytime-valid, but drops the proxy.Naive peeking (
PPIMeanEstimatorper batch) targets the debiased true mean as well, but drops anytime-validity.
Naive peeking is statistically invalid by construction and is included for comparison only.
import matplotlib.pyplot as plt
import numpy as np
from glide.estimators import ClassicalMeanEstimator, PPIMeanEstimator
from glide.monitors import ClassicalMeanMonitor, PPIMeanMonitor
from glide.samplers import StratifiedSampler
from glide.scientific_validation import coverage_with_error_bar
from glide.simulators import generate_stratified_binary_dataset, simulate_annotation
plt.rcParams.update(
{
"font.size": 18,
"axes.labelsize": 18,
"axes.titlesize": 18,
"legend.fontsize": 16,
"xtick.labelsize": 16,
"ytick.labelsize": 16,
"figure.titlesize": 19,
}
)
Experiment Parameters¶
We fix every parameter up front. Each stratum of the simulated dataset is treated as one chronologically monitored batch; the strata array is used directly as batches.
N_BATCHES,BATCH_SIZE,LABELS_PER_BATCH: stream length, number of samples per batch, and human-labeled samples per batch (allocated withStratifiedSampler).TRUE_MEAN,PROXY_BIAS: the stationary true risk and the fixed proxy bias, soproxy_mean = true_mean + PROXY_BIASat every batch and the proxy always co-moves with the true risk, before and after any drift.CORRELATION: the fixed proxy/true correlation, reflecting an LLM judge's quality.THRESHOLD: the alarm boundary, the highest tolerated risk value.DRIFT_START,AMPLITUDES: for the abrupt-drift workflow only, the batch at whichtrue_meanandproxy_meanstep up together, and the grid of step sizes swept in the detection-power figure.CONFIDENCE_LEVEL: the confidence level used throughout (1 - confidence_levelis the total false-alarm budget).N_SEEDS,ERROR_BAR_CONFIDENCE_LEVEL: the number of Monte Carlo streams, and the confidence level of the error bars drawn on every estimated rate or delay.
N_BATCHES = 100
BATCH_SIZE = 160
LABELS_PER_BATCH = 40
TRUE_MEAN = 0.3
PROXY_BIAS = 0.05
CORRELATION = 0.85
THRESHOLD = 0.35
DRIFT_START = 25
AMPLITUDES = np.array([0.125, 0.15, 0.175, 0.2, 0.225])
CONFIDENCE_LEVEL = 0.8
N_SEEDS = 250
ERROR_BAR_CONFIDENCE_LEVEL = 0.9
METHODS = ["PPRM", "Classical", "Naive (peeking)"]
VALID_METHODS = ["PPRM", "Classical"]
METHOD_COLORS = {"PPRM": "darkorange", "Classical": "steelblue", "Naive (peeking)": "red"}
Data Generation¶
The functions in the next cell use generate_stratified_binary_dataset to generate simulated data for two scenarios:
- No drift : every batch's true and proxy labels are drawn from Bernoulli distributions with a constant mean.
- Abrupt drift : batches before
DRIFT_STARTfollow the same non-alarming distribution; batches fromDRIFT_STARTonward follow a distribution with a higher mean risk, above the alarm threshold.
Both scenarios first generate true and proxy labels for every instance, then keep a fixed number of true labels per batch and mask the rest, to simulate what would be observed in practice.
def simulate_no_drift_stream(seed):
true_means = np.full(N_BATCHES, TRUE_MEAN)
proxy_means = true_means + PROXY_BIAS
correlations = np.full(N_BATCHES, CORRELATION)
y_true_oracle, y_proxy, batches = generate_stratified_binary_dataset(
n_total=np.full(N_BATCHES, BATCH_SIZE),
true_mean=true_means,
proxy_mean=proxy_means,
correlation=correlations,
random_seed=seed,
)
xi = StratifiedSampler().sample(
y_proxy, groups=batches, n_samples=LABELS_PER_BATCH * N_BATCHES, strategy="proportional", random_seed=seed
)
y_true = simulate_annotation(y_true_oracle, xi)
return y_true, y_proxy, batches
def simulate_abrupt_drift_stream(seed, amplitude):
true_means = np.full(N_BATCHES, TRUE_MEAN)
true_means[DRIFT_START:] += amplitude
proxy_means = true_means + PROXY_BIAS
correlations = np.full(N_BATCHES, CORRELATION)
y_true_oracle, y_proxy, batches = generate_stratified_binary_dataset(
n_total=np.full(N_BATCHES, BATCH_SIZE),
true_mean=true_means,
proxy_mean=proxy_means,
correlation=correlations,
random_seed=seed,
)
xi = StratifiedSampler().sample(
y_proxy, groups=batches, n_samples=LABELS_PER_BATCH * N_BATCHES, strategy="proportional", random_seed=seed
)
y_true = simulate_annotation(y_true_oracle, xi)
return y_true, y_proxy, batches
A single stream, for intuition¶
Before running Monte Carlo experiments, we simulate one abrupt-drift stream to show what the monitors return: a running mean and an anytime-valid lower bound on it, batch by batch.
AMPLITUDE_EXAMPLE = 0.15
y_true_example, y_proxy_example, batches_example = simulate_abrupt_drift_stream(seed=8, amplitude=AMPLITUDE_EXAMPLE)
pprm_example = PPIMeanMonitor().detect(
y_true_example,
y_proxy_example,
batches_example,
higher_is_better=False,
threshold=THRESHOLD,
confidence_level=CONFIDENCE_LEVEL,
)
classical_example = ClassicalMeanMonitor().detect(
y_true_example,
batches_example,
higher_is_better=False,
threshold=THRESHOLD,
confidence_level=CONFIDENCE_LEVEL,
)
Let us plot the computed lower bounds along with the true risk before and after the drift.
fig, ax = plt.subplots(figsize=(9, 5))
batch_axis = np.arange(1, N_BATCHES + 1)
for name, result in [("PPRM", pprm_example), ("Classical", classical_example)]:
color = METHOD_COLORS[name]
ax.plot(batch_axis, result.confidence_bounds, color=color, label=name)
exceeds_threshold = np.where(result.confidence_bounds > THRESHOLD)[0]
if exceeds_threshold.size > 0:
first_exceed_batch = batch_axis[exceeds_threshold[0]]
ax.axvline(first_exceed_batch, color=color, linestyle=":", lw=1.5)
ax.plot(
[0, DRIFT_START, DRIFT_START, N_BATCHES],
2 * [TRUE_MEAN] + 2 * [TRUE_MEAN + AMPLITUDE_EXAMPLE],
color="dimgray",
linestyle="-.",
lw=1.5,
label="True risk",
)
ax.axhline(THRESHOLD, color="black", linestyle="--", lw=1.5, label="Threshold")
ax.axvline(DRIFT_START, color="gray", linestyle=":", lw=1.5, label="Drift start")
ax.set_xlabel("Batches")
ax.set_ylabel("Risk lower bound")
ax.legend()
plt.tight_layout()
plt.show()
The bound climbs after the drift starts and eventually crosses the threshold, at which point detection occurs. The Monte Carlo experiments below repeat this once per seed to characterize false-alarm and detection behavior across the whole distribution of streams.
The Monte Carlo Loop¶
The functions below implement each of the three monitoring methods. The naive baseline calls PPIMeanEstimator on each batch in isolation and alarms the moment a single batch's confidence interval lies entirely above the threshold: this is the repeated-testing procedure described in the user guide. The second one runs all three methods on one instance of generated data and returns their first alarm indices.
A stream that never alarms is given a sentinel first alarm index of N_BATCHES (one past the last valid index), so it registers as not-yet-alarmed at every batch, without a separate case to track.
def naive_first_alarm_index(y_true, y_proxy, batches, confidence_level, threshold):
n_batches = len(np.unique(batches))
for batch_id in range(n_batches):
batch_mask = batches == batch_id
batch_result = PPIMeanEstimator().estimate(
y_true[batch_mask], y_proxy[batch_mask], confidence_level=confidence_level
)
if batch_result.confidence_interval.lower_bound > threshold:
return batch_id
return n_batches
def run_all_methods(y_true, y_proxy, batches, confidence_level, threshold):
pprm_result = PPIMeanMonitor().detect(
y_true, y_proxy, batches, higher_is_better=False, threshold=threshold, confidence_level=confidence_level
)
classical_result = ClassicalMeanMonitor().detect(
y_true, batches, higher_is_better=False, threshold=threshold, confidence_level=confidence_level
)
naive_index = naive_first_alarm_index(y_true, y_proxy, batches, confidence_level, threshold)
pprm_index = pprm_result.first_alarm_index if pprm_result.drift_detected else N_BATCHES
classical_index = classical_result.first_alarm_index if classical_result.drift_detected else N_BATCHES
return {"PPRM": pprm_index, "Classical": classical_index, "Naive (peeking)": naive_index}
A single external loop over the seeds calls both workflows once per seed, using the functions above to get each method's first alarm index. For the no-drift workflow, a single stream is generated per seed. For the abrupt-drift workflow, each amplitude in AMPLITUDES requires its own stream, so data is regenerated per amplitude. The two workflows do not share data with each other.
no_drift_first_alarms = {method: np.empty(N_SEEDS, dtype=int) for method in METHODS}
abrupt_drift_first_alarms = {
amplitude: {method: np.empty(N_SEEDS, dtype=int) for method in METHODS} for amplitude in AMPLITUDES
}
for seed in range(N_SEEDS):
y_true, y_proxy, batches = simulate_no_drift_stream(seed)
outcomes = run_all_methods(y_true, y_proxy, batches, confidence_level=CONFIDENCE_LEVEL, threshold=THRESHOLD)
for method in METHODS:
no_drift_first_alarms[method][seed] = outcomes[method]
for amplitude in AMPLITUDES:
y_true, y_proxy, batches = simulate_abrupt_drift_stream(seed, amplitude)
outcomes = run_all_methods(y_true, y_proxy, batches, confidence_level=CONFIDENCE_LEVEL, threshold=THRESHOLD)
for method in METHODS:
abrupt_drift_first_alarms[amplitude][method][seed] = outcomes[method]
False-Alarm Control Under No Drift¶
The stream is stationary: all batches are sampled from the same distribution whose risk is below the threshold, so no method should alarm except by chance. Checking a fresh confidence interval after every batch, as the naive baseline does, is a repeated-testing (peeking) failure: even though every individual look was calibrated, the repeated tests are almost sure to raise a false alarm at a long enough horizon. An anytime-valid confidence sequence, in contrast, bounds the probability of ever alarming over the whole horizon by the single budget.
Cumulative false-alarm rate¶
For each batch, the plot below shows the empirical probability that a stream has alarmed by that point: the fraction of the simulated streams that have already raised an alarm, at the default confidence level. Because a stream that has alarmed stays alarmed, this curve is the empirical cumulative distribution of the first alarm index, non-decreasing over batches.
batch_axis = np.arange(1, N_BATCHES + 1)
budget = 1 - CONFIDENCE_LEVEL
fig, ax = plt.subplots(figsize=(9, 5))
for method in METHODS:
first_alarms = no_drift_first_alarms[method]
means, lowers, uppers = np.zeros(len(batch_axis)), np.zeros(len(batch_axis)), np.zeros(len(batch_axis))
for i in range(N_BATCHES):
hits = (first_alarms <= batch_axis[i] - 1).astype(float)
mean_rate, lower, upper = coverage_with_error_bar(hits, ERROR_BAR_CONFIDENCE_LEVEL)
means[i] = mean_rate
lowers[i] = lower
uppers[i] = upper
color = METHOD_COLORS[method]
ax.plot(batch_axis, means, color=color, label=method)
ax.fill_between(batch_axis, lowers, uppers, alpha=0.15, color=color)
ax.axhline(budget, color="black", linestyle="--", lw=2, label="False-alarm budget")
ax.set_xlabel("Batches observed")
ax.set_ylabel("Alarm rate")
ax.legend()
plt.tight_layout()
plt.show()
The naive baseline's curve climbs steadily, as each additional batch adds another chance to false-alarm. It crosses the budget line before the end of the horizon: peeking is unsafe. PPRM and Classical both stay flat and near zero for every batch, including the terminal one. Their anytime-valid guarantee bounds the probability of ever alarming over the whole horizon. The empirical-Bernstein construction that provides it is conservative, so the observed rate typically sits well below the nominal budget.
Detection Power and Speed Under Abrupt Drift¶
The stream now steps at batch DRIFT_START: the true and proxy means both jump from their stationary values to a higher risk level, keeping the same fixed correlation, for the remaining batches. Because PPRM's estimand is the debiased true mean, only a genuine shift in the true mean constitutes drift; the proxy is kept co-moving purely so it stays informative. AMPLITUDES sweeps the size of this jump, and for each amplitude and seed we record whether each method ever alarms and, if so, at what delay.
Only PPRM and Classical are compared here: a delay axis is not meaningful for the naive baseline, which alarms early for the wrong reason (peeking rather than genuine evidence) and whose invalidity is established above.
fig, (ax_rate, ax_delay) = plt.subplots(1, 2, figsize=(14, 6))
for method in VALID_METHODS:
color = METHOD_COLORS[method]
plotted_amplitudes, rate_means, rate_lowers, rate_uppers = [], [], [], []
delay_means, delay_lowers, delay_uppers = [], [], []
for amplitude in AMPLITUDES:
first_alarms = abrupt_drift_first_alarms[amplitude][method]
detection_mask = first_alarms < N_BATCHES
detected = first_alarms[detection_mask]
detection_rate, rate_lower, rate_upper = coverage_with_error_bar(
detection_mask.astype(float), ERROR_BAR_CONFIDENCE_LEVEL
)
delay_result = ClassicalMeanEstimator().estimate(
detected.astype(float), confidence_level=ERROR_BAR_CONFIDENCE_LEVEL
)
plotted_amplitudes.append(amplitude)
rate_means.append(detection_rate)
rate_lowers.append(rate_lower)
rate_uppers.append(rate_upper)
delay_means.append(delay_result.mean)
delay_lowers.append(delay_result.confidence_interval.lower_bound)
delay_uppers.append(delay_result.confidence_interval.upper_bound)
ax_rate.plot(plotted_amplitudes, rate_means, marker="o", color=color, label=method)
ax_rate.fill_between(plotted_amplitudes, rate_lowers, rate_uppers, alpha=0.15, color=color)
ax_delay.plot(plotted_amplitudes, delay_means, marker="o", color=color, label=method)
ax_delay.fill_between(plotted_amplitudes, delay_lowers, delay_uppers, alpha=0.15, color=color)
ax_rate.set_xlabel("Amplitude")
ax_rate.set_ylabel("Detection rate")
ax_rate.legend()
ax_delay.set_xlabel("Amplitude")
ax_delay.set_ylabel("Mean detection delay")
ax_delay.legend()
plt.tight_layout()
plt.show()
The left panel plots detection rate against amplitude; the right panel plots mean detection delay. Both improve as amplitude grows: detection rate rises and mean detection delay falls. PPRM detects more reliably and faster than Classical across the amplitudes tested.
Summary¶
This notebook has empirically validated that GLIDE's PPRM implementation satisfies both properties required of a valid monitor:
| Method | Valid under repeated looks? | Detects abrupt drift? | Detection delay, relative to Classical |
|---|---|---|---|
| PPRM | Yes | Yes | Shorter |
| Classical | Yes | Yes | Baseline |
| Naive peeking | No | Yes, but invalidly | Not comparable (invalid) |
Naive peeking is invalid for continuous risk monitoring: checking a fresh confidence interval after every batch inevitably exceeds the false-alarm budget at long horizons. Both PPRM and Classical control the false-alarm rate at every confidence level, and PPRM additionally leverages proxy labels to detect drift more reliably and faster.