Scientific Validity of PPI++ for Mean Estimation¶
This notebook provides empirical evidence that GLIDE's Prediction-Powered Inference (PPI++) implementation is statistically valid.
Setup: We estimate the mean of a binary outcome (e.g., the hallucination rate of an AI system). We have:
- A small set of true labels (
y_true), expensive but unbiased - A large set of proxy labels (
y_proxy), cheap but potentially biased
PPI++ combines both to produce confidence intervals that are:
- Valid : they cover the true mean at the specified rate (e.g. 90% confidence)
- Shorter : as compared to those obtained with true labels only, especially when the proxy is strongly correlated with the truth
We test these two claims empirically across a range of proxy/true correlation levels.
from functools import partial
import matplotlib.pyplot as plt
import numpy as np
from glide.estimators import ClassicalMeanEstimator, PPIMeanEstimator
from glide.samplers import UniformSampler
from glide.scientific_validation import compute_hits, coverage_with_error_bar, run_monte_carlo
from glide.simulators import generate_binary_dataset, simulate_annotation
plt.rcParams.update(
{
"font.size": 18,
"axes.labelsize": 18,
"axes.titlesize": 18,
"legend.fontsize": 16,
"xtick.labelsize": 16,
"ytick.labelsize": 16,
"figure.titlesize": 19,
}
)
Experiment Parameters¶
We fix all parameters up front so every section of this notebook uses a consistent setup. We define :
CONFIDENCE_LEVEL: the confidence level at which we will compute confidence intervals.N_TOTAL: the total number of samples (human annotated + proxy labeled).BUDGET: the number of human annotated samples.TRUE_MEAN: the true mean value of human labels.PROXY_MEAN: the (biased) proxy mean value;N_SEEDS: the number of simulations we will make in our Monte Carlo experiments.
Note on correlation bounds: Depending on the values of
TRUE_MEANandPROXY_MEAN, extreme correlation values (close to -1 or 1) may not be possible. Correlation sweeps are kept within these limits.
Finally, we define the baseline estimation methods for comparison:
True only: refers to using true human labels only to compute estimates in the conventional wayProxy only: refers to using proxy labels only.PPI++: refers to Prediction Power Inference with power-tuning, details are provided below.
CONFIDENCE_LEVEL = 0.9
N_TOTAL = 1500
BUDGET = 500
TRUE_MEAN = 0.55
PROXY_MEAN = 0.5
N_SEEDS = 1000
METHODS = ["True only", "Proxy only", "PPI++"]
correlations = np.arange(0.1, 0.95, 0.1)
n_correlations = len(correlations)
correlations_lmh = [
correlations[n_correlations // 4],
correlations[n_correlations // 2],
correlations[3 * n_correlations // 4],
] # low, medium and high values
corr_labels = ["Low", "Medium", "High"]
Data Simulation¶
We use generate_binary_dataset to simulate a realistic evaluation scenario. It simulates correlated binary labels (y_true_oracle, y_proxy) for all N_TOTAL samples. The absence of certain ground-truths is then simulated by randomly selecting BUDGET samples to annotate via UniformSampler and masking the rest with np.nan via simulate_annotation.
The correlation parameter controls the Pearson correlation between true and proxy labels.
# Single example dataset for illustration
y_true_oracle, y_proxy = generate_binary_dataset(
n_total=N_TOTAL,
true_mean=TRUE_MEAN,
proxy_mean=PROXY_MEAN,
correlation=0.8,
random_seed=42,
)
xi = UniformSampler().sample(n_samples=N_TOTAL, budget=BUDGET, random_seed=42)
y_true = simulate_annotation(y_true_oracle, xi)
Inference Results¶
We compare three estimation methods:
| Estimation method | Data used | Notes |
|---|---|---|
| True only | y_true |
Classical CLT Confidence Interval, the gold standard for validity |
| Proxy only | y_proxy |
Biased, cheap but wrong |
| PPI++ | y_true + y_proxy (rectified) |
Best of both worlds, valid and efficient |
The function below simulates a dataset for a given seed and correlation level, then runs all three estimation methods on it.
def simulate_estimates(seed, correlation):
y_true, y_proxy = generate_binary_dataset(
n_total=N_TOTAL,
true_mean=TRUE_MEAN,
proxy_mean=PROXY_MEAN,
correlation=correlation,
random_seed=seed,
)
xi = UniformSampler().sample(N_TOTAL, BUDGET, random_seed=seed)
y_true = simulate_annotation(y_true, xi)
# --- PPI++ ---
estimator = PPIMeanEstimator()
ppi_result = estimator.estimate(y_true, y_proxy, confidence_level=CONFIDENCE_LEVEL)
# --- Classical baselines ---
classical_estimator = ClassicalMeanEstimator()
true_only_result = classical_estimator.estimate(y_true, confidence_level=CONFIDENCE_LEVEL)
proxy_only_result = classical_estimator.estimate(y_proxy, confidence_level=CONFIDENCE_LEVEL)
return {
"True only": {
"mean": true_only_result.mean,
"std": true_only_result.std,
"confidence_interval": true_only_result.confidence_interval,
},
"Proxy only": {
"mean": proxy_only_result.mean,
"std": proxy_only_result.std,
"confidence_interval": proxy_only_result.confidence_interval,
},
"PPI++": {
"mean": ppi_result.mean,
"std": ppi_result.std,
"confidence_interval": ppi_result.confidence_interval,
"effective_sample_size": ppi_result.effective_sample_size,
},
}
PPI++ is implemented by the PPIMeanEstimator whereas ClassicalMeanEstimator implements conventional mean estimation.
Coverage Validity¶
A confidence interval is valid if it reliably captures the true value at the nominal rate: a 90% confidence interval is valid if, across many repetitions, around 90% of the resulting intervals contain the true value.
We run a Monte Carlo experiment to verify this for each method. We check that the empirical coverage tracks the nominal level throughout. See the Scientific Validation Methodology page for more details about the verification protocol.
Coverage vs confidence level for three correlation levels¶
We sweep the confidence level from 0.55 to 0.95 and plot the observed coverage. For a valid estimation method, the dots should fall on or above the black diagonal $y = \text{confidence level}$.
We do this for low, medium and high proxy correlation.
# Run Monte Carlo simulations for each correlation level
confidence_levels = np.arange(0.55, 1.00, 0.05)
confidence_levels = np.round(confidence_levels, 2)
raw_stats = {
corr: run_monte_carlo(confidence_levels, partial(simulate_estimates, correlation=corr)) for corr in correlations
}
# Derive coverage for every (correlation, confidence_level) pair
coverages_confidence_intervals = {}
for correlation in correlations_lmh:
coverages_confidence_intervals[correlation] = {}
for confidence_level in confidence_levels:
hits = compute_hits(raw_stats[correlation], confidence_level, TRUE_MEAN)
coverages_confidence_intervals[correlation][confidence_level] = dict()
for method in METHODS:
coverage_confidence_interval = coverage_with_error_bar(hits[method], confidence_level=CONFIDENCE_LEVEL)
coverages_confidence_intervals[correlation][confidence_level][method] = coverage_confidence_interval
fig, axes = plt.subplots(1, 3, figsize=(15, 5), sharey=True)
colors = {"True only": "steelblue", "PPI++": "darkorange", "Proxy only": "red"}
for ax, correlation, label in zip(axes, correlations_lmh, corr_labels):
ax.plot(confidence_levels, confidence_levels, color="black", lw=1.5, linestyle="--", label="Ideal")
for method in METHODS:
mean_ci = np.array([coverages_confidence_intervals[correlation][cl][method] for cl in confidence_levels])
mean = mean_ci[:, 0]
lo = mean_ci[:, 1]
hi = mean_ci[:, 2]
ax.plot(confidence_levels, mean, marker="o", color=colors[method], label=method)
ax.fill_between(confidence_levels, lo, hi, alpha=0.15, color=colors[method])
ax.set_title(f"{label} correlation (${round(correlation, 2)}$)")
ax.set_xlabel("Target confidence level")
ax.set_ylabel("Observed coverage")
ax.legend()
ax.set_xlim(0.5, 1.0)
ax.set_ylim(0.5, 1.0)
plt.tight_layout()
plt.show()
Both PPI++ and True only track the diagonal closely across all correlation levels, confirming that PPI++ achieves valid coverage regardless of proxy quality. The Proxy only method does not show up because it uses biased data so that its coverage is invalid (close to zero).
Coverage vs correlation for fixed confidence level¶
We now fix the confidence level and sweep a range of proxy-true correlation levels. This shows that PPI++' validity does not degrade as the proxy becomes weaker.
coverage_by_corr = {} # {correlation: {method: observed mean coverage}}
coverage_ci_by_corr = {} # {correlation: {method: (lower, upper) Confidence Interval on coverage}}
for correlation in correlations:
hits = compute_hits(raw_stats[correlation], CONFIDENCE_LEVEL, TRUE_MEAN)
coverage_by_corr[correlation] = {}
coverage_ci_by_corr[correlation] = {}
for method in METHODS:
mean_cov, lo, hi = coverage_with_error_bar(hits[method], CONFIDENCE_LEVEL)
coverage_by_corr[correlation][method] = mean_cov
coverage_ci_by_corr[correlation][method] = (lo, hi)
fig, ax = plt.subplots(figsize=(8, 5))
method_colors = {"True only": "steelblue", "PPI++": "darkorange"}
for method in ["True only", "PPI++"]:
obs = np.array([coverage_by_corr[correlation][method] for correlation in correlations])
ci_bounds = np.array([coverage_ci_by_corr[correlation][method] for correlation in correlations])
lo = ci_bounds[:, 0]
hi = ci_bounds[:, 1]
ax.plot(correlations, obs, marker="o", color=method_colors[method], label=method)
ax.fill_between(correlations, lo, hi, alpha=0.15, color=method_colors[method])
ax.axhline(y=CONFIDENCE_LEVEL, color="red", linestyle="--", lw=2, label=f"Target coverage {CONFIDENCE_LEVEL:.0%}")
ax.set_xlabel("Proxy–true correlation")
ax.set_ylabel("Observed coverage")
ax.set_xlim(0, 1)
ax.set_ylim(0.8, 1.0)
ax.yaxis.set_ticks(ax.get_yticks()[1:-1:2])
ax.legend()
plt.tight_layout()
plt.show()
Note that Proxy only is not plotted because the proxy is biased (proxy mean ≠ true mean). Therefore it has invalid coverage (close to 0) whereas PPI++ and True only remain valid across all correlation levels.
Confidence Interval Width¶
Coverage validity is necessary but not sufficient, we also want short intervals. PPI++'s promise is that by leveraging the unlabeled proxy data, it remains statistically valid, just like using labeled data alone, but with a shorter interval when the proxy is informative.
We compare mean confidence interval widths for PPI and True only across correlation levels.
width_by_corr = {}
for correlation in correlations:
width_by_corr[correlation] = {}
for method in METHODS:
lower_bound = raw_stats[correlation][method]["lower_bounds"][CONFIDENCE_LEVEL]
upper_bound = raw_stats[correlation][method]["upper_bounds"][CONFIDENCE_LEVEL]
width_by_corr[correlation][method] = upper_bound - lower_bound
fig, ax = plt.subplots(figsize=(9, 5))
plot_methods = ["True only", "PPI++"]
colors_w = {"True only": "steelblue", "PPI++": "darkorange"}
# Compute percentiles based on CONFIDENCE_LEVEL
lower_percentile = round(((1 - CONFIDENCE_LEVEL) / 2) * 100)
upper_percentile = 100 - lower_percentile
for method in plot_methods:
means_w = [np.mean(width_by_corr[correlation][method]) for correlation in correlations]
q_lower = [np.percentile(width_by_corr[correlation][method], lower_percentile) for correlation in correlations]
q_upper = [np.percentile(width_by_corr[correlation][method], upper_percentile) for correlation in correlations]
ax.plot(correlations, means_w, marker="o", label=method, color=colors_w[method])
ax.fill_between(correlations, q_lower, q_upper, alpha=0.15, color=colors_w[method])
ax.set_xlabel("Proxy–true correlation")
ax.set_ylabel("Confidence Interval width")
ax.set_xlim(0.05, 0.95)
ax.legend()
plt.tight_layout()
plt.show()
As expected, PPI++' interval width decreases with increasing correlation. Leveraging the unlabeled proxy data is only beneficial when the proxy is informative.
Effective Sample Size¶
A natural summary of PPI's efficiency gain is the effective sample size (ESS): the number of true labels that would be needed to match PPI's mean confidence interval width.
We report PPI's effective sample size across correlation levels, translating the width reduction into an equivalent number of true labels. See the Scientific Validation Methodology page for the formal definition and formula of ESS.
ess_mean = [np.mean(raw_stats[correlation]["PPI++"]["effective_sample_sizes"]) for correlation in correlations]
ess_q_lower = [
np.percentile(raw_stats[correlation]["PPI++"]["effective_sample_sizes"], lower_percentile)
for correlation in correlations
]
ess_q_upper = [
np.percentile(raw_stats[correlation]["PPI++"]["effective_sample_sizes"], upper_percentile)
for correlation in correlations
]
fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(correlations, ess_mean, marker="o", color="darkorange", label="PPI++ ESS (mean)")
ax.fill_between(
correlations,
ess_q_lower,
ess_q_upper,
alpha=0.15,
color="darkorange",
label=f"{lower_percentile:.0f}th–{upper_percentile:.0f}th percentile",
)
ax.axhline(y=BUDGET, color="steelblue", linestyle="--", lw=2, label=f"Baseline (True only, n={BUDGET})")
ax.set_xlabel("Proxy–true correlation")
ax.set_ylabel("Effective sample size")
ax.set_xlim(0.05, 0.95)
ax.legend()
plt.tight_layout()
plt.show()
Summary¶
This notebook has empirically validated that GLIDE's PPI++ implementation satisfies two key statistical properties:
| Property | Result |
|---|---|
| Coverage validity | PPI++ achieves the nominal coverage across all correlation levels and confidence levels tested |
| Efficiency | PPI++ produces shorter confidence intervals than labeled-only whenever correlation is positive, with the gain growing quadratically with correlation |
Crucially, the biased baseline (Proxy only) fails the coverage test. It appears precise but is systematically wrong. PPI++ avoids this by correcting for proxy bias using the labeled subset.
The ESS analysis shows that with a proxy correlation of $0.9$, PPI++ is equivalent to having roughly twice more labeled data, a significant practical gain in scenarios where true annotation is expensive. This highlights the importance of a good LLM judge to evaluate an AI system.