Skip to content

Monitors

glide.monitors.asymptotic_classical.AsymptoticClassicalRM

Bases: AsymptoticRM[NDArray, None]

Anytime-valid label-only drift monitor leveraging each batch estimate's standard deviation.

It computes a per-batch sample mean of the labeled values, then tracks the running mean of those estimates against a user-supplied threshold. The comparison uses a Gaussian-mixture asymptotic confidence sequence whose width scales with the standard error of each batch mean, so genuine drifts get flagged early. The false-alarm guarantee is asymptotic: each batch needs enough labeled samples (preferably tens) for its sample mean to be approximately Gaussian with a consistently estimated variance.

References

Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya Ramdas. "Time-uniform central limit theory and asymptotic confidence sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

Examples:

>>> import numpy as np
>>> from glide.monitors import AsymptoticClassicalRM
>>> pre_drift_batch = np.array([0.0, 0.2, np.nan, np.nan])
>>> post_drift_batch = np.array([0.8, 1.0, np.nan, np.nan])
>>> y = np.hstack([pre_drift_batch, np.tile(post_drift_batch, 5)])
>>> batches = np.repeat(np.arange(6), 4)
>>> monitor = AsymptoticClassicalRM()
>>> result = monitor.detect(y, batches, higher_is_better=False, threshold=0.5)
>>> result.drift_detected
True
>>> result.first_alarm_index
3
Source code in glide/monitors/asymptotic_classical.py 
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
class AsymptoticClassicalRM(AsymptoticRM[NDArray, None]):
    """Anytime-valid label-only drift monitor leveraging each batch estimate's
    standard deviation.

    It computes a per-batch sample mean of the labeled values, then tracks the
    running mean of those estimates against a user-supplied ``threshold``. The
    comparison uses a Gaussian-mixture asymptotic confidence sequence whose
    width scales with the standard error of each batch mean, so genuine drifts
    get flagged early. The false-alarm guarantee is asymptotic: each batch
    needs enough labeled samples (preferably tens) for its sample mean to be
    approximately Gaussian with a consistently estimated variance.

    References
    ----------
    Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya
    Ramdas. "Time-uniform central limit theory and asymptotic confidence
    sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

    Examples
    --------
    >>> import numpy as np
    >>> from glide.monitors import AsymptoticClassicalRM
    >>> pre_drift_batch = np.array([0.0, 0.2, np.nan, np.nan])
    >>> post_drift_batch = np.array([0.8, 1.0, np.nan, np.nan])
    >>> y = np.hstack([pre_drift_batch, np.tile(post_drift_batch, 5)])
    >>> batches = np.repeat(np.arange(6), 4)
    >>> monitor = AsymptoticClassicalRM()
    >>> result = monitor.detect(y, batches, higher_is_better=False, threshold=0.5)
    >>> result.drift_detected
    True
    >>> result.first_alarm_index
    3
    """

    _engine = ClassicalMeanEngine()

    def detect(
        self,
        y: NDArray,
        batches: NDArray,
        *,
        higher_is_better: bool,
        threshold: float,
        metric_name: str = "Metric",
        confidence_level: float = 0.8,
        tightest_at_batch: int = 10,
    ) -> ClassicalMeanMonitoringResult:
        """Detect a drift of the running mean across a batched dataset.

        Splits the data by batch, computes a sample-mean estimate and its standard
        error per batch, and builds an anytime-valid asymptotic confidence sequence
        on the running mean of those estimates. An alarm is raised at every batch
        where the sequence crosses the user-supplied ``threshold``.

        Rows must be ordered oldest batch first and grouped into contiguous blocks;
        identifier values are not compared, so any label type works. Batches must be
        non-overlapping (no shared samples), and successive calls must be made on
        growing histories of the same data; passing the full accumulated dataset at
        every call makes the anytime-valid guarantee hold jointly over all calls.
        Alternatively the data may be restricted to the most recent batches, in which
        case the guarantee holds within each restriction but not across the moving
        history as a whole. When the history has fewer batches than ``tightest_at_batch``,
        the tuning target is set to the last batch. Prefix-consistency only holds between
        calls using the same ``tightest_at_batch`` value and with histories containing
        at least ``tightest_at_batch`` batches.

        Parameters
        ----------
        y : NDArray
            Array of observations, shape ``(n_samples,)``. Unlabeled entries are
            ``np.nan`` and are dropped per batch; every batch must keep at least 2
            labeled entries, though tens are recommended in practice for the
            asymptotic guarantee to hold.
        batches : NDArray
            Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
            oldest batch first and grouped into contiguous blocks. Identifier values
            are not compared, so any hashable label type works (integers, dates,
            free-form strings).
        higher_is_better : bool
            ``False`` when the metric is a risk (drift means the metric increased),
            ``True`` when it is a performance (drift means the metric decreased).
        threshold : float
            The metric value the running mean is monitored against, in metric units:
            the worst level the user is willing to tolerate. An alarm fires once the
            anytime-valid bound proves the running metric has crossed it (the running
            risk exceeds it for a risk, the running performance falls below it for a
            performance).
        metric_name : str, optional
            Human-readable label for the metric. Defaults to ``"Metric"``.
        confidence_level : float, optional
            How confident each alarm should be. At the default ``0.8`` the monitor
            raises "80% confident" alarms: under no drift it has at most a 20% chance
            of raising a false alarm. Raising this value demands more evidence before
            alarming, so alarms become more trustworthy but arrive later; lowering it
            detects sooner at the cost of more false alarms. Must be in ``(0.5, 1)``.
        tightest_at_batch : int, optional
            The batch index (1-indexed) at which the confidence sequence is tuned to
            be tightest. Defaults to ``10``. Only affects tightness, not validity:
            bounds stay anytime-valid at every batch regardless of this choice.

        Returns
        -------
        ClassicalMeanMonitoringResult
            Per-batch estimates, running means, anytime-valid confidence bounds,
            alarm flags, and the alarm threshold, all in the original metric
            orientation.

        Raises
        ------
        ValueError
            - If ``batches`` is empty.
            - If ``y`` and ``batches`` have different lengths.
            - If ``batches`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
            - If ``confidence_level`` is not in ``(0.5, 1)``.
            - If batches are interleaved rather than grouped into contiguous blocks.
            - If any batch has fewer than 2 labeled (non-NaN) samples.
            - If ``tightest_at_batch`` is not a positive integer.
            - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
        """
        batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
            label_fields={"y": y},
            batches=batches,
            higher_is_better=higher_is_better,
            confidence_level=confidence_level,
            tightest_at_batch=tightest_at_batch,
            power_tuning=False,
        )
        confidence_sequence = AsymptoticConfidenceSequence(
            running_mean_estimates=running_means, confidence_bounds=confidence_bounds
        )
        labeled_mask = ~np.isnan(y)
        batch_n = np.bincount(batch_codes[labeled_mask])
        result = ClassicalMeanMonitoringResult(
            metric_name=metric_name,
            monitor_name=self.__class__.__name__,
            higher_is_better=higher_is_better,
            alarm_threshold=threshold,
            confidence_level=confidence_level,
            batch_mean_estimates=batch_mean_estimates,
            confidence_sequence=confidence_sequence,
            batch_n=batch_n,
        )
        return result

detect 

detect(
    y,
    batches,
    *,
    higher_is_better,
    threshold,
    metric_name="Metric",
    confidence_level=0.8,
    tightest_at_batch=10,
)

Detect a drift of the running mean across a batched dataset.

Splits the data by batch, computes a sample-mean estimate and its standard error per batch, and builds an anytime-valid asymptotic confidence sequence on the running mean of those estimates. An alarm is raised at every batch where the sequence crosses the user-supplied threshold.

Rows must be ordered oldest batch first and grouped into contiguous blocks; identifier values are not compared, so any label type works. Batches must be non-overlapping (no shared samples), and successive calls must be made on growing histories of the same data; passing the full accumulated dataset at every call makes the anytime-valid guarantee hold jointly over all calls. Alternatively the data may be restricted to the most recent batches, in which case the guarantee holds within each restriction but not across the moving history as a whole. When the history has fewer batches than tightest_at_batch, the tuning target is set to the last batch. Prefix-consistency only holds between calls using the same tightest_at_batch value and with histories containing at least tightest_at_batch batches.

Parameters:

Name Type Description Default
y NDArray

Array of observations, shape (n_samples,). Unlabeled entries are np.nan and are dropped per batch; every batch must keep at least 2 labeled entries, though tens are recommended in practice for the asymptotic guarantee to hold.

 required
batches NDArray

Array of batch identifiers, shape (n_samples,). Rows must be ordered oldest batch first and grouped into contiguous blocks. Identifier values are not compared, so any hashable label type works (integers, dates, free-form strings).

 required
higher_is_better bool

False when the metric is a risk (drift means the metric increased), True when it is a performance (drift means the metric decreased).

 required
threshold float

The metric value the running mean is monitored against, in metric units: the worst level the user is willing to tolerate. An alarm fires once the anytime-valid bound proves the running metric has crossed it (the running risk exceeds it for a risk, the running performance falls below it for a performance).

 required
metric_name str

Human-readable label for the metric. Defaults to "Metric".

 'Metric'
confidence_level float

How confident each alarm should be. At the default 0.8 the monitor raises "80% confident" alarms: under no drift it has at most a 20% chance of raising a false alarm. Raising this value demands more evidence before alarming, so alarms become more trustworthy but arrive later; lowering it detects sooner at the cost of more false alarms. Must be in (0.5, 1).

 0.8
tightest_at_batch int

The batch index (1-indexed) at which the confidence sequence is tuned to be tightest. Defaults to 10. Only affects tightness, not validity: bounds stay anytime-valid at every batch regardless of this choice.

 10

Returns:

Type Description
ClassicalMeanMonitoringResult

Per-batch estimates, running means, anytime-valid confidence bounds, alarm flags, and the alarm threshold, all in the original metric orientation.

Raises:

Type Description
ValueError
  • If batches is empty.
  • If y and batches have different lengths.
  • If batches contains NaN values (numeric dtype) or None values (non-numeric dtype).
  • If confidence_level is not in (0.5, 1).
  • If batches are interleaved rather than grouped into contiguous blocks.
  • If any batch has fewer than 2 labeled (non-NaN) samples.
  • If tightest_at_batch is not a positive integer.
  • If the accumulated variance of the batch estimates up to tightest_at_batch is zero.
Source code in glide/monitors/asymptotic_classical.py 
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
def detect(
    self,
    y: NDArray,
    batches: NDArray,
    *,
    higher_is_better: bool,
    threshold: float,
    metric_name: str = "Metric",
    confidence_level: float = 0.8,
    tightest_at_batch: int = 10,
) -> ClassicalMeanMonitoringResult:
    """Detect a drift of the running mean across a batched dataset.

    Splits the data by batch, computes a sample-mean estimate and its standard
    error per batch, and builds an anytime-valid asymptotic confidence sequence
    on the running mean of those estimates. An alarm is raised at every batch
    where the sequence crosses the user-supplied ``threshold``.

    Rows must be ordered oldest batch first and grouped into contiguous blocks;
    identifier values are not compared, so any label type works. Batches must be
    non-overlapping (no shared samples), and successive calls must be made on
    growing histories of the same data; passing the full accumulated dataset at
    every call makes the anytime-valid guarantee hold jointly over all calls.
    Alternatively the data may be restricted to the most recent batches, in which
    case the guarantee holds within each restriction but not across the moving
    history as a whole. When the history has fewer batches than ``tightest_at_batch``,
    the tuning target is set to the last batch. Prefix-consistency only holds between
    calls using the same ``tightest_at_batch`` value and with histories containing
    at least ``tightest_at_batch`` batches.

    Parameters
    ----------
    y : NDArray
        Array of observations, shape ``(n_samples,)``. Unlabeled entries are
        ``np.nan`` and are dropped per batch; every batch must keep at least 2
        labeled entries, though tens are recommended in practice for the
        asymptotic guarantee to hold.
    batches : NDArray
        Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
        oldest batch first and grouped into contiguous blocks. Identifier values
        are not compared, so any hashable label type works (integers, dates,
        free-form strings).
    higher_is_better : bool
        ``False`` when the metric is a risk (drift means the metric increased),
        ``True`` when it is a performance (drift means the metric decreased).
    threshold : float
        The metric value the running mean is monitored against, in metric units:
        the worst level the user is willing to tolerate. An alarm fires once the
        anytime-valid bound proves the running metric has crossed it (the running
        risk exceeds it for a risk, the running performance falls below it for a
        performance).
    metric_name : str, optional
        Human-readable label for the metric. Defaults to ``"Metric"``.
    confidence_level : float, optional
        How confident each alarm should be. At the default ``0.8`` the monitor
        raises "80% confident" alarms: under no drift it has at most a 20% chance
        of raising a false alarm. Raising this value demands more evidence before
        alarming, so alarms become more trustworthy but arrive later; lowering it
        detects sooner at the cost of more false alarms. Must be in ``(0.5, 1)``.
    tightest_at_batch : int, optional
        The batch index (1-indexed) at which the confidence sequence is tuned to
        be tightest. Defaults to ``10``. Only affects tightness, not validity:
        bounds stay anytime-valid at every batch regardless of this choice.

    Returns
    -------
    ClassicalMeanMonitoringResult
        Per-batch estimates, running means, anytime-valid confidence bounds,
        alarm flags, and the alarm threshold, all in the original metric
        orientation.

    Raises
    ------
    ValueError
        - If ``batches`` is empty.
        - If ``y`` and ``batches`` have different lengths.
        - If ``batches`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
        - If ``confidence_level`` is not in ``(0.5, 1)``.
        - If batches are interleaved rather than grouped into contiguous blocks.
        - If any batch has fewer than 2 labeled (non-NaN) samples.
        - If ``tightest_at_batch`` is not a positive integer.
        - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
    """
    batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
        label_fields={"y": y},
        batches=batches,
        higher_is_better=higher_is_better,
        confidence_level=confidence_level,
        tightest_at_batch=tightest_at_batch,
        power_tuning=False,
    )
    confidence_sequence = AsymptoticConfidenceSequence(
        running_mean_estimates=running_means, confidence_bounds=confidence_bounds
    )
    labeled_mask = ~np.isnan(y)
    batch_n = np.bincount(batch_codes[labeled_mask])
    result = ClassicalMeanMonitoringResult(
        metric_name=metric_name,
        monitor_name=self.__class__.__name__,
        higher_is_better=higher_is_better,
        alarm_threshold=threshold,
        confidence_level=confidence_level,
        batch_mean_estimates=batch_mean_estimates,
        confidence_sequence=confidence_sequence,
        batch_n=batch_n,
    )
    return result

glide.monitors.asymptotic_clustered_classical.AsymptoticClusteredClassicalRM

Bases: AsymptoticRM[NDArray, None]

Anytime-valid label-only drift monitor over clustered data.

Computes a per-batch estimate by reducing each batch's labeled observations to per-cluster means and taking the classical mean of those means, together with its standard error, then tracks the running mean of the per-batch estimates the same way AsymptoticClassicalRM does for independent, unclustered samples. Samples of the same cluster are correlated, so treating each cluster as a single sampling unit keeps the standard errors, and hence the false-alarm guarantee, valid.

References

Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya Ramdas. "Time-uniform central limit theory and asymptotic confidence sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

Examples:

>>> import numpy as np
>>> from glide.monitors import AsymptoticClusteredClassicalRM
>>> pre_drift_batch = np.array([0.0, 0.2, 0.1, 0.3, np.nan, np.nan])
>>> post_drift_batch = np.array([0.8, 1.0, 0.7, 0.9, np.nan, np.nan])
>>> y = np.hstack([pre_drift_batch, np.tile(post_drift_batch, 5)])
>>> clusters = np.repeat(np.arange(18), 2)
>>> batches = np.repeat(np.arange(6), 6)
>>> monitor = AsymptoticClusteredClassicalRM()
>>> result = monitor.detect(y, clusters, batches, higher_is_better=False, threshold=0.5)
>>> result.drift_detected
True
>>> result.first_alarm_index
2
Source code in glide/monitors/asymptotic_clustered_classical.py 
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
class AsymptoticClusteredClassicalRM(AsymptoticRM[NDArray, None]):
    """Anytime-valid label-only drift monitor over clustered data.

    Computes a per-batch estimate by reducing each batch's labeled observations
    to per-cluster means and taking the classical mean of those means, together
    with its standard error, then tracks the running mean of the per-batch
    estimates the same way ``AsymptoticClassicalRM`` does for independent,
    unclustered samples. Samples of the same cluster are correlated, so treating
    each cluster as a single sampling unit keeps the standard errors, and hence
    the false-alarm guarantee, valid.

    References
    ----------
    Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya
    Ramdas. "Time-uniform central limit theory and asymptotic confidence
    sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

    Examples
    --------
    >>> import numpy as np
    >>> from glide.monitors import AsymptoticClusteredClassicalRM
    >>> pre_drift_batch = np.array([0.0, 0.2, 0.1, 0.3, np.nan, np.nan])
    >>> post_drift_batch = np.array([0.8, 1.0, 0.7, 0.9, np.nan, np.nan])
    >>> y = np.hstack([pre_drift_batch, np.tile(post_drift_batch, 5)])
    >>> clusters = np.repeat(np.arange(18), 2)
    >>> batches = np.repeat(np.arange(6), 6)
    >>> monitor = AsymptoticClusteredClassicalRM()
    >>> result = monitor.detect(y, clusters, batches, higher_is_better=False, threshold=0.5)
    >>> result.drift_detected
    True
    >>> result.first_alarm_index
    2
    """

    _engine = ClusteredClassicalMeanEngine()

    def detect(
        self,
        y: NDArray,
        clusters: NDArray,
        batches: NDArray,
        *,
        higher_is_better: bool,
        threshold: float,
        metric_name: str = "Metric",
        confidence_level: float = 0.8,
        tightest_at_batch: int = 10,
    ) -> ClassicalMeanMonitoringResult:
        """Detect a drift of the running mean across a batched, clustered dataset.

        Splits the data by batch, reduces each batch's labeled observations to
        per-cluster means, computes the classical mean and its standard error on
        those means, and builds an anytime-valid asymptotic confidence sequence on
        the running mean of the per-batch estimates. An alarm is raised at every
        batch where the sequence crosses the user-supplied ``threshold``. See
        ``AsymptoticClassicalRM.detect`` for the full explanation of batch ordering,
        growing-history semantics, and ``tightest_at_batch``, which all carry over
        unchanged.

        Parameters
        ----------
        y : NDArray
            Array of observations, shape ``(n_samples,)``. Unlabeled entries are
            ``np.nan`` and are dropped per batch; every batch must keep labeled
            entries in at least 2 clusters, though tens are recommended in practice
            for the asymptotic guarantee to hold.
        clusters : NDArray
            Array of cluster identifiers, shape ``(n_samples,)``. Any hashable label
            type works (integers, strings, ...). Must be unique across the whole
            stream: an identifier reused in two different batches raises, since
            their rows would otherwise be silently pooled into a single artificial
            cluster once both fall inside the same prefix.
        batches : NDArray
            Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
            oldest batch first and grouped into contiguous blocks. Identifier values
            are not compared, so any hashable label type works (integers, dates,
            free-form strings).
        higher_is_better : bool
            ``False`` when the metric is a risk (drift means the metric increased),
            ``True`` when it is a performance (drift means the metric decreased).
        threshold : float
            The metric value the running mean is monitored against, in metric units:
            the worst level the user is willing to tolerate. An alarm fires once the
            anytime-valid bound proves the running metric has crossed it (the running
            risk exceeds it for a risk, the running performance falls below it for a
            performance).
        metric_name : str, optional
            Human-readable label for the metric. Defaults to ``"Metric"``.
        confidence_level : float, optional
            How confident each alarm should be. At the default ``0.8`` the monitor
            raises "80% confident" alarms: under no drift it has at most a 20% chance
            of raising a false alarm. Raising this value demands more evidence before
            alarming, so alarms become more trustworthy but arrive later; lowering it
            detects sooner at the cost of more false alarms. Must be in ``(0.5, 1)``.
        tightest_at_batch : int, optional
            The batch index (1-indexed) at which the confidence sequence is tuned to
            be tightest. Defaults to ``10``. Only affects tightness, not validity:
            bounds stay anytime-valid at every batch regardless of this choice.

        Returns
        -------
        ClassicalMeanMonitoringResult
            Per-batch estimates, running means, anytime-valid confidence bounds,
            alarm flags, and the alarm threshold, all in the original metric
            orientation. ``batch_n`` counts the labeled observations of each batch,
            aggregated across its clusters.

        Raises
        ------
        ValueError
            - If ``batches`` is empty.
            - If ``y``, ``clusters`` and ``batches`` have different lengths.
            - If ``batches`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
            - If ``clusters`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
            - If ``confidence_level`` is not in ``(0.5, 1)``.
            - If batches are interleaved rather than grouped into contiguous blocks.
            - If any batch has labeled observations in fewer than 2 clusters.
            - If a cluster identifier's rows span more than one batch.
            - If ``tightest_at_batch`` is not a positive integer.
            - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
        """
        _validate_clusters_unique_across_batches(clusters, batches)
        batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
            label_fields={"y": y},
            identifier_fields={"clusters": clusters},
            batches=batches,
            higher_is_better=higher_is_better,
            confidence_level=confidence_level,
            tightest_at_batch=tightest_at_batch,
            power_tuning=False,
        )
        confidence_sequence = AsymptoticConfidenceSequence(
            running_mean_estimates=running_means, confidence_bounds=confidence_bounds
        )
        labeled_mask = ~np.isnan(y)
        batch_n = np.bincount(batch_codes[labeled_mask])
        result = ClassicalMeanMonitoringResult(
            metric_name=metric_name,
            monitor_name=self.__class__.__name__,
            higher_is_better=higher_is_better,
            alarm_threshold=threshold,
            confidence_level=confidence_level,
            batch_mean_estimates=batch_mean_estimates,
            confidence_sequence=confidence_sequence,
            batch_n=batch_n,
        )
        return result

detect 

detect(
    y,
    clusters,
    batches,
    *,
    higher_is_better,
    threshold,
    metric_name="Metric",
    confidence_level=0.8,
    tightest_at_batch=10,
)

Detect a drift of the running mean across a batched, clustered dataset.

Splits the data by batch, reduces each batch's labeled observations to per-cluster means, computes the classical mean and its standard error on those means, and builds an anytime-valid asymptotic confidence sequence on the running mean of the per-batch estimates. An alarm is raised at every batch where the sequence crosses the user-supplied threshold. See AsymptoticClassicalRM.detect for the full explanation of batch ordering, growing-history semantics, and tightest_at_batch, which all carry over unchanged.

Parameters:

Name Type Description Default
y NDArray

Array of observations, shape (n_samples,). Unlabeled entries are np.nan and are dropped per batch; every batch must keep labeled entries in at least 2 clusters, though tens are recommended in practice for the asymptotic guarantee to hold.

 required
clusters NDArray

Array of cluster identifiers, shape (n_samples,). Any hashable label type works (integers, strings, ...). Must be unique across the whole stream: an identifier reused in two different batches raises, since their rows would otherwise be silently pooled into a single artificial cluster once both fall inside the same prefix.

 required
batches NDArray

Array of batch identifiers, shape (n_samples,). Rows must be ordered oldest batch first and grouped into contiguous blocks. Identifier values are not compared, so any hashable label type works (integers, dates, free-form strings).

 required
higher_is_better bool

False when the metric is a risk (drift means the metric increased), True when it is a performance (drift means the metric decreased).

 required
threshold float

The metric value the running mean is monitored against, in metric units: the worst level the user is willing to tolerate. An alarm fires once the anytime-valid bound proves the running metric has crossed it (the running risk exceeds it for a risk, the running performance falls below it for a performance).

 required
metric_name str

Human-readable label for the metric. Defaults to "Metric".

 'Metric'
confidence_level float

How confident each alarm should be. At the default 0.8 the monitor raises "80% confident" alarms: under no drift it has at most a 20% chance of raising a false alarm. Raising this value demands more evidence before alarming, so alarms become more trustworthy but arrive later; lowering it detects sooner at the cost of more false alarms. Must be in (0.5, 1).

 0.8
tightest_at_batch int

The batch index (1-indexed) at which the confidence sequence is tuned to be tightest. Defaults to 10. Only affects tightness, not validity: bounds stay anytime-valid at every batch regardless of this choice.

 10

Returns:

Type Description
ClassicalMeanMonitoringResult

Per-batch estimates, running means, anytime-valid confidence bounds, alarm flags, and the alarm threshold, all in the original metric orientation. batch_n counts the labeled observations of each batch, aggregated across its clusters.

Raises:

Type Description
ValueError
  • If batches is empty.
  • If y, clusters and batches have different lengths.
  • If batches contains NaN values (numeric dtype) or None values (non-numeric dtype).
  • If clusters contains NaN values (numeric dtype) or None values (non-numeric dtype).
  • If confidence_level is not in (0.5, 1).
  • If batches are interleaved rather than grouped into contiguous blocks.
  • If any batch has labeled observations in fewer than 2 clusters.
  • If a cluster identifier's rows span more than one batch.
  • If tightest_at_batch is not a positive integer.
  • If the accumulated variance of the batch estimates up to tightest_at_batch is zero.
Source code in glide/monitors/asymptotic_clustered_classical.py 
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
def detect(
    self,
    y: NDArray,
    clusters: NDArray,
    batches: NDArray,
    *,
    higher_is_better: bool,
    threshold: float,
    metric_name: str = "Metric",
    confidence_level: float = 0.8,
    tightest_at_batch: int = 10,
) -> ClassicalMeanMonitoringResult:
    """Detect a drift of the running mean across a batched, clustered dataset.

    Splits the data by batch, reduces each batch's labeled observations to
    per-cluster means, computes the classical mean and its standard error on
    those means, and builds an anytime-valid asymptotic confidence sequence on
    the running mean of the per-batch estimates. An alarm is raised at every
    batch where the sequence crosses the user-supplied ``threshold``. See
    ``AsymptoticClassicalRM.detect`` for the full explanation of batch ordering,
    growing-history semantics, and ``tightest_at_batch``, which all carry over
    unchanged.

    Parameters
    ----------
    y : NDArray
        Array of observations, shape ``(n_samples,)``. Unlabeled entries are
        ``np.nan`` and are dropped per batch; every batch must keep labeled
        entries in at least 2 clusters, though tens are recommended in practice
        for the asymptotic guarantee to hold.
    clusters : NDArray
        Array of cluster identifiers, shape ``(n_samples,)``. Any hashable label
        type works (integers, strings, ...). Must be unique across the whole
        stream: an identifier reused in two different batches raises, since
        their rows would otherwise be silently pooled into a single artificial
        cluster once both fall inside the same prefix.
    batches : NDArray
        Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
        oldest batch first and grouped into contiguous blocks. Identifier values
        are not compared, so any hashable label type works (integers, dates,
        free-form strings).
    higher_is_better : bool
        ``False`` when the metric is a risk (drift means the metric increased),
        ``True`` when it is a performance (drift means the metric decreased).
    threshold : float
        The metric value the running mean is monitored against, in metric units:
        the worst level the user is willing to tolerate. An alarm fires once the
        anytime-valid bound proves the running metric has crossed it (the running
        risk exceeds it for a risk, the running performance falls below it for a
        performance).
    metric_name : str, optional
        Human-readable label for the metric. Defaults to ``"Metric"``.
    confidence_level : float, optional
        How confident each alarm should be. At the default ``0.8`` the monitor
        raises "80% confident" alarms: under no drift it has at most a 20% chance
        of raising a false alarm. Raising this value demands more evidence before
        alarming, so alarms become more trustworthy but arrive later; lowering it
        detects sooner at the cost of more false alarms. Must be in ``(0.5, 1)``.
    tightest_at_batch : int, optional
        The batch index (1-indexed) at which the confidence sequence is tuned to
        be tightest. Defaults to ``10``. Only affects tightness, not validity:
        bounds stay anytime-valid at every batch regardless of this choice.

    Returns
    -------
    ClassicalMeanMonitoringResult
        Per-batch estimates, running means, anytime-valid confidence bounds,
        alarm flags, and the alarm threshold, all in the original metric
        orientation. ``batch_n`` counts the labeled observations of each batch,
        aggregated across its clusters.

    Raises
    ------
    ValueError
        - If ``batches`` is empty.
        - If ``y``, ``clusters`` and ``batches`` have different lengths.
        - If ``batches`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
        - If ``clusters`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
        - If ``confidence_level`` is not in ``(0.5, 1)``.
        - If batches are interleaved rather than grouped into contiguous blocks.
        - If any batch has labeled observations in fewer than 2 clusters.
        - If a cluster identifier's rows span more than one batch.
        - If ``tightest_at_batch`` is not a positive integer.
        - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
    """
    _validate_clusters_unique_across_batches(clusters, batches)
    batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
        label_fields={"y": y},
        identifier_fields={"clusters": clusters},
        batches=batches,
        higher_is_better=higher_is_better,
        confidence_level=confidence_level,
        tightest_at_batch=tightest_at_batch,
        power_tuning=False,
    )
    confidence_sequence = AsymptoticConfidenceSequence(
        running_mean_estimates=running_means, confidence_bounds=confidence_bounds
    )
    labeled_mask = ~np.isnan(y)
    batch_n = np.bincount(batch_codes[labeled_mask])
    result = ClassicalMeanMonitoringResult(
        metric_name=metric_name,
        monitor_name=self.__class__.__name__,
        higher_is_better=higher_is_better,
        alarm_threshold=threshold,
        confidence_level=confidence_level,
        batch_mean_estimates=batch_mean_estimates,
        confidence_sequence=confidence_sequence,
        batch_n=batch_n,
    )
    return result

glide.monitors.asymptotic_pprm.AsymptoticPPRM

Bases: AsymptoticRM[PPIDataset, float]

Anytime-valid drift monitor leveraging the variance of each batch estimate.

It computes a per-batch Prediction-Powered Inference (PPI) estimate: a small set of true labels and a large set of proxy labels, combined with a power-tuning parameter fitted on the batches that strictly precede it (predictable, as required for validity), together with the standard error of that estimate. The monitor tracks the running mean of the per-batch estimates against a user-supplied threshold, through a Gaussian-mixture asymptotic confidence sequence whose width scales with the standard error of each batch estimate, so genuine drifts get flagged early. The false-alarm guarantee is asymptotic: each batch needs enough labeled and proxy samples for its PPI estimate to be approximately Gaussian with a consistently estimated variance.

References

Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya Ramdas. "Time-uniform central limit theory and asymptotic confidence sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

Zhang, Guangyi, Yunlong Cai, Guanding Yu, and Osvaldo Simeone. "Prediction-powered risk monitoring of deployed models for detecting harmful distribution shifts." arXiv preprint arXiv:2602.02229 (2026).

Podkopaev, Aleksandr, and Aaditya Ramdas. "Tracking the risk of a deployed model and detecting harmful distribution shifts." International Conference on Learning Representations (ICLR), 2022.

Examples:

>>> import numpy as np
>>> from glide.monitors import AsymptoticPPRM
>>> pre_drift_y_true = np.array([0.0, 0.2, np.nan, np.nan])
>>> pre_drift_y_proxy = np.array([0.0, 0.2, 0.0, 0.2])
>>> post_drift_y_true = np.array([0.8, 1.0, np.nan, np.nan])
>>> post_drift_y_proxy = np.array([0.8, 1.0, 0.8, 1.0])
>>> y_true = np.hstack([pre_drift_y_true, np.tile(post_drift_y_true, 5)])
>>> y_proxy = np.hstack([pre_drift_y_proxy, np.tile(post_drift_y_proxy, 5)])
>>> batches = np.repeat(np.arange(6), 4)
>>> monitor = AsymptoticPPRM()
>>> result = monitor.detect(y_true, y_proxy, batches, higher_is_better=False, threshold=0.5)
>>> result.drift_detected
True
>>> result.first_alarm_index
2
Source code in glide/monitors/asymptotic_pprm.py 
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
class AsymptoticPPRM(AsymptoticRM[PPIDataset, float]):
    """Anytime-valid drift monitor leveraging the variance of each batch estimate.

    It computes a per-batch Prediction-Powered Inference (PPI) estimate: a small set
    of true labels and a large set of proxy labels, combined with a power-tuning
    parameter fitted on the batches that strictly precede it (predictable, as
    required for validity), together with the standard error of that estimate. The
    monitor tracks the running mean of the per-batch estimates against a
    user-supplied ``threshold``, through a Gaussian-mixture asymptotic confidence
    sequence whose width scales with the standard error of each batch estimate, so
    genuine drifts get flagged early. The false-alarm guarantee is asymptotic: each
    batch needs enough labeled and proxy samples for its PPI estimate to be
    approximately Gaussian with a consistently estimated variance.

    References
    ----------
    Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya
    Ramdas. "Time-uniform central limit theory and asymptotic confidence
    sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

    Zhang, Guangyi, Yunlong Cai, Guanding Yu, and Osvaldo Simeone. "Prediction-powered
    risk monitoring of deployed models for detecting harmful distribution shifts."
    arXiv preprint arXiv:2602.02229 (2026).

    Podkopaev, Aleksandr, and Aaditya Ramdas. "Tracking the risk of a deployed model
    and detecting harmful distribution shifts." International Conference on Learning
    Representations (ICLR), 2022.

    Examples
    --------
    >>> import numpy as np
    >>> from glide.monitors import AsymptoticPPRM
    >>> pre_drift_y_true = np.array([0.0, 0.2, np.nan, np.nan])
    >>> pre_drift_y_proxy = np.array([0.0, 0.2, 0.0, 0.2])
    >>> post_drift_y_true = np.array([0.8, 1.0, np.nan, np.nan])
    >>> post_drift_y_proxy = np.array([0.8, 1.0, 0.8, 1.0])
    >>> y_true = np.hstack([pre_drift_y_true, np.tile(post_drift_y_true, 5)])
    >>> y_proxy = np.hstack([pre_drift_y_proxy, np.tile(post_drift_y_proxy, 5)])
    >>> batches = np.repeat(np.arange(6), 4)
    >>> monitor = AsymptoticPPRM()
    >>> result = monitor.detect(y_true, y_proxy, batches, higher_is_better=False, threshold=0.5)
    >>> result.drift_detected
    True
    >>> result.first_alarm_index
    2
    """

    _engine = PPIMeanEngine()

    def detect(
        self,
        y_true: NDArray,
        y_proxy: NDArray,
        batches: NDArray,
        *,
        higher_is_better: bool,
        threshold: float,
        metric_name: str = "Metric",
        confidence_level: float = 0.8,
        power_tuning: bool = True,
        tightest_at_batch: int = 10,
    ) -> PredictionPoweredMeanMonitoringResult:
        """Detect a drift of the running mean across a batched dataset.

        Splits the data by batch, computes a prediction-powered estimate and its
        standard error per batch, and builds an anytime-valid asymptotic confidence
        sequence on the running mean of those estimates. An alarm is raised at every
        batch where the sequence crosses the user-supplied ``threshold``.

        Rows must be ordered oldest batch first and grouped into contiguous blocks;
        identifier values are not compared, so any label type works. Batches must be
        non-overlapping (no shared samples), and successive calls must be made on
        growing histories of the same data; passing the full accumulated dataset at
        every call makes the anytime-valid guarantee hold jointly over all calls.
        Alternatively the data may be restricted to the most recent batches, in which
        case the guarantee holds within each restriction but not across the moving
        history as a whole. When the history has fewer batches than ``tightest_at_batch``,
        the tuning target is set to the last batch. Prefix-consistency only holds between
        calls using the same ``tightest_at_batch`` value and with histories containing
        at least ``tightest_at_batch`` batches.

        Parameters
        ----------
        y_true : NDArray
            Array of labeled observations, shape ``(n_samples,)``.
            Labeled entries are finite; unlabeled entries are ``np.nan``.
        y_proxy : NDArray
            Array of proxy predictions, shape ``(n_samples,)``.
            Must be fully populated (no NaN).
        batches : NDArray
            Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
            oldest batch first and grouped into contiguous blocks. Identifier values
            are not compared, so any hashable label type works (integers, dates,
            free-form strings).
        higher_is_better : bool
            ``False`` when the metric is a risk (drift means the metric increased),
            ``True`` when it is a performance (drift means the metric decreased).
        threshold : float
            The metric value the running mean is monitored against, in metric units:
            the worst level the user is willing to tolerate. An alarm fires once the
            anytime-valid bound proves the running metric has crossed it (the running
            risk exceeds it for a risk, the running performance falls below it for a
            performance).
        metric_name : str, optional
            Human-readable label for the metric. Defaults to ``"Metric"``.
        confidence_level : float, optional
            How confident each alarm should be. At the default ``0.8`` the monitor
            raises "80% confident" alarms: under no drift it has at most a 20% chance
            of raising a false alarm. Raising this value demands more evidence before
            alarming, so alarms become more trustworthy but arrive later; lowering it
            detects sooner at the cost of more false alarms. Must be in ``(0.5, 1)``.
        power_tuning : bool, optional
            If ``True`` (default), compute the power-tuning parameter of each batch on
            all previous batches (the first batch, having no predecessor, uses
            ``1.0``). If ``False``, use ``1.0`` everywhere (classic PPI).
        tightest_at_batch : int, optional
            The batch index (1-indexed) at which the confidence sequence is tuned to
            be tightest. Defaults to ``10``, which is safe to leave alone: this
            setting affects tightness only, never the false-alarm guarantee, and the
            width penalty for being off by a factor of 10 is below 10%.

        Returns
        -------
        PredictionPoweredMeanMonitoringResult
            Per-batch estimates, running means, anytime-valid confidence bounds,
            alarm flags, and the alarm threshold, all in the original metric
            orientation.

        Raises
        ------
        ValueError
            - If ``batches`` is empty.
            - If ``y_true``, ``y_proxy`` and ``batches`` have different lengths.
            - If ``batches`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
            - If ``confidence_level`` is not in ``(0.5, 1)``.
            - If any proxy value is NaN.
            - If batches are interleaved rather than grouped into contiguous blocks.
            - If any batch has fewer than 2 labeled or fewer than 2 unlabeled samples.
            - If proxy values are constant across a prefix set of batches (with ``power_tuning=True``).
            - If ``tightest_at_batch`` is not a positive integer.
            - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
        """
        batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
            label_fields={"y_true": y_true, "y_proxy": y_proxy},
            batches=batches,
            higher_is_better=higher_is_better,
            confidence_level=confidence_level,
            tightest_at_batch=tightest_at_batch,
            power_tuning=power_tuning,
        )
        confidence_sequence = AsymptoticConfidenceSequence(
            running_mean_estimates=running_means, confidence_bounds=confidence_bounds
        )
        labeled_mask = ~np.isnan(y_true)
        batch_n_true = np.bincount(batch_codes[labeled_mask])
        batch_n_proxy = np.bincount(batch_codes)
        result = PredictionPoweredMeanMonitoringResult(
            metric_name=metric_name,
            monitor_name=self.__class__.__name__,
            higher_is_better=higher_is_better,
            alarm_threshold=threshold,
            confidence_level=confidence_level,
            batch_mean_estimates=batch_mean_estimates,
            confidence_sequence=confidence_sequence,
            batch_n_true=batch_n_true,
            batch_n_proxy=batch_n_proxy,
        )
        return result

detect 

detect(
    y_true,
    y_proxy,
    batches,
    *,
    higher_is_better,
    threshold,
    metric_name="Metric",
    confidence_level=0.8,
    power_tuning=True,
    tightest_at_batch=10,
)

Detect a drift of the running mean across a batched dataset.

Splits the data by batch, computes a prediction-powered estimate and its standard error per batch, and builds an anytime-valid asymptotic confidence sequence on the running mean of those estimates. An alarm is raised at every batch where the sequence crosses the user-supplied threshold.

Rows must be ordered oldest batch first and grouped into contiguous blocks; identifier values are not compared, so any label type works. Batches must be non-overlapping (no shared samples), and successive calls must be made on growing histories of the same data; passing the full accumulated dataset at every call makes the anytime-valid guarantee hold jointly over all calls. Alternatively the data may be restricted to the most recent batches, in which case the guarantee holds within each restriction but not across the moving history as a whole. When the history has fewer batches than tightest_at_batch, the tuning target is set to the last batch. Prefix-consistency only holds between calls using the same tightest_at_batch value and with histories containing at least tightest_at_batch batches.

Parameters:

Name Type Description Default
y_true NDArray

Array of labeled observations, shape (n_samples,). Labeled entries are finite; unlabeled entries are np.nan.

 required
y_proxy NDArray

Array of proxy predictions, shape (n_samples,). Must be fully populated (no NaN).

 required
batches NDArray

Array of batch identifiers, shape (n_samples,). Rows must be ordered oldest batch first and grouped into contiguous blocks. Identifier values are not compared, so any hashable label type works (integers, dates, free-form strings).

 required
higher_is_better bool

False when the metric is a risk (drift means the metric increased), True when it is a performance (drift means the metric decreased).

 required
threshold float

The metric value the running mean is monitored against, in metric units: the worst level the user is willing to tolerate. An alarm fires once the anytime-valid bound proves the running metric has crossed it (the running risk exceeds it for a risk, the running performance falls below it for a performance).

 required
metric_name str

Human-readable label for the metric. Defaults to "Metric".

 'Metric'
confidence_level float

How confident each alarm should be. At the default 0.8 the monitor raises "80% confident" alarms: under no drift it has at most a 20% chance of raising a false alarm. Raising this value demands more evidence before alarming, so alarms become more trustworthy but arrive later; lowering it detects sooner at the cost of more false alarms. Must be in (0.5, 1).

 0.8
power_tuning bool

If True (default), compute the power-tuning parameter of each batch on all previous batches (the first batch, having no predecessor, uses 1.0). If False, use 1.0 everywhere (classic PPI).

 True
tightest_at_batch int

The batch index (1-indexed) at which the confidence sequence is tuned to be tightest. Defaults to 10, which is safe to leave alone: this setting affects tightness only, never the false-alarm guarantee, and the width penalty for being off by a factor of 10 is below 10%.

 10

Returns:

Type Description
PredictionPoweredMeanMonitoringResult

Per-batch estimates, running means, anytime-valid confidence bounds, alarm flags, and the alarm threshold, all in the original metric orientation.

Raises:

Type Description
ValueError
  • If batches is empty.
  • If y_true, y_proxy and batches have different lengths.
  • If batches contains NaN values (numeric dtype) or None values (non-numeric dtype).
  • If confidence_level is not in (0.5, 1).
  • If any proxy value is NaN.
  • If batches are interleaved rather than grouped into contiguous blocks.
  • If any batch has fewer than 2 labeled or fewer than 2 unlabeled samples.
  • If proxy values are constant across a prefix set of batches (with power_tuning=True).
  • If tightest_at_batch is not a positive integer.
  • If the accumulated variance of the batch estimates up to tightest_at_batch is zero.
Source code in glide/monitors/asymptotic_pprm.py 
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
def detect(
    self,
    y_true: NDArray,
    y_proxy: NDArray,
    batches: NDArray,
    *,
    higher_is_better: bool,
    threshold: float,
    metric_name: str = "Metric",
    confidence_level: float = 0.8,
    power_tuning: bool = True,
    tightest_at_batch: int = 10,
) -> PredictionPoweredMeanMonitoringResult:
    """Detect a drift of the running mean across a batched dataset.

    Splits the data by batch, computes a prediction-powered estimate and its
    standard error per batch, and builds an anytime-valid asymptotic confidence
    sequence on the running mean of those estimates. An alarm is raised at every
    batch where the sequence crosses the user-supplied ``threshold``.

    Rows must be ordered oldest batch first and grouped into contiguous blocks;
    identifier values are not compared, so any label type works. Batches must be
    non-overlapping (no shared samples), and successive calls must be made on
    growing histories of the same data; passing the full accumulated dataset at
    every call makes the anytime-valid guarantee hold jointly over all calls.
    Alternatively the data may be restricted to the most recent batches, in which
    case the guarantee holds within each restriction but not across the moving
    history as a whole. When the history has fewer batches than ``tightest_at_batch``,
    the tuning target is set to the last batch. Prefix-consistency only holds between
    calls using the same ``tightest_at_batch`` value and with histories containing
    at least ``tightest_at_batch`` batches.

    Parameters
    ----------
    y_true : NDArray
        Array of labeled observations, shape ``(n_samples,)``.
        Labeled entries are finite; unlabeled entries are ``np.nan``.
    y_proxy : NDArray
        Array of proxy predictions, shape ``(n_samples,)``.
        Must be fully populated (no NaN).
    batches : NDArray
        Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
        oldest batch first and grouped into contiguous blocks. Identifier values
        are not compared, so any hashable label type works (integers, dates,
        free-form strings).
    higher_is_better : bool
        ``False`` when the metric is a risk (drift means the metric increased),
        ``True`` when it is a performance (drift means the metric decreased).
    threshold : float
        The metric value the running mean is monitored against, in metric units:
        the worst level the user is willing to tolerate. An alarm fires once the
        anytime-valid bound proves the running metric has crossed it (the running
        risk exceeds it for a risk, the running performance falls below it for a
        performance).
    metric_name : str, optional
        Human-readable label for the metric. Defaults to ``"Metric"``.
    confidence_level : float, optional
        How confident each alarm should be. At the default ``0.8`` the monitor
        raises "80% confident" alarms: under no drift it has at most a 20% chance
        of raising a false alarm. Raising this value demands more evidence before
        alarming, so alarms become more trustworthy but arrive later; lowering it
        detects sooner at the cost of more false alarms. Must be in ``(0.5, 1)``.
    power_tuning : bool, optional
        If ``True`` (default), compute the power-tuning parameter of each batch on
        all previous batches (the first batch, having no predecessor, uses
        ``1.0``). If ``False``, use ``1.0`` everywhere (classic PPI).
    tightest_at_batch : int, optional
        The batch index (1-indexed) at which the confidence sequence is tuned to
        be tightest. Defaults to ``10``, which is safe to leave alone: this
        setting affects tightness only, never the false-alarm guarantee, and the
        width penalty for being off by a factor of 10 is below 10%.

    Returns
    -------
    PredictionPoweredMeanMonitoringResult
        Per-batch estimates, running means, anytime-valid confidence bounds,
        alarm flags, and the alarm threshold, all in the original metric
        orientation.

    Raises
    ------
    ValueError
        - If ``batches`` is empty.
        - If ``y_true``, ``y_proxy`` and ``batches`` have different lengths.
        - If ``batches`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
        - If ``confidence_level`` is not in ``(0.5, 1)``.
        - If any proxy value is NaN.
        - If batches are interleaved rather than grouped into contiguous blocks.
        - If any batch has fewer than 2 labeled or fewer than 2 unlabeled samples.
        - If proxy values are constant across a prefix set of batches (with ``power_tuning=True``).
        - If ``tightest_at_batch`` is not a positive integer.
        - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
    """
    batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
        label_fields={"y_true": y_true, "y_proxy": y_proxy},
        batches=batches,
        higher_is_better=higher_is_better,
        confidence_level=confidence_level,
        tightest_at_batch=tightest_at_batch,
        power_tuning=power_tuning,
    )
    confidence_sequence = AsymptoticConfidenceSequence(
        running_mean_estimates=running_means, confidence_bounds=confidence_bounds
    )
    labeled_mask = ~np.isnan(y_true)
    batch_n_true = np.bincount(batch_codes[labeled_mask])
    batch_n_proxy = np.bincount(batch_codes)
    result = PredictionPoweredMeanMonitoringResult(
        metric_name=metric_name,
        monitor_name=self.__class__.__name__,
        higher_is_better=higher_is_better,
        alarm_threshold=threshold,
        confidence_level=confidence_level,
        batch_mean_estimates=batch_mean_estimates,
        confidence_sequence=confidence_sequence,
        batch_n_true=batch_n_true,
        batch_n_proxy=batch_n_proxy,
    )
    return result

glide.monitors.asymptotic_stratified_pprm.AsymptoticStratifiedPPRM

Bases: AsymptoticRM[StratifiedPPIDataset, StratifiedTuningParameter]

Anytime-valid drift monitor combining human and proxy labels over a stratified stream.

Computes a per-batch Stratified PPI estimate together with its standard error. Each batch's samples are split into strata, a power-tuned PPI++ estimate is computed independently within each stratum using a weight fitted on the batches that strictly precede it, and the per-stratum estimates are combined with population-proportional weights. The monitor then tracks the running mean of the per-batch estimates the same way AsymptoticPPRM does for an unstratified stream. The false-alarm guarantee is asymptotic: each batch needs enough labeled and proxy samples, within each of its strata, for its Stratified PPI estimate to be approximately Gaussian with a consistently estimated variance.

References

Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya Ramdas. "Time-uniform central limit theory and asymptotic confidence sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

Zhang, Guangyi, Yunlong Cai, Guanding Yu, and Osvaldo Simeone. "Prediction-powered risk monitoring of deployed models for detecting harmful distribution shifts." arXiv preprint arXiv:2602.02229 (2026).

Podkopaev, Aleksandr, and Aaditya Ramdas. "Tracking the risk of a deployed model and detecting harmful distribution shifts." International Conference on Learning Representations (ICLR), 2022.

Fisch, Adam, Joshua Maynez, R. Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William W. Cohen. "Stratified prediction-powered inference for effective hybrid evaluation of language models." Advances in Neural Information Processing Systems 37 (2024): 111489-111514.

Fogliato, Riccardo, Pratik Patil, Mathew Monfort, and Pietro Perona. "A framework for efficient model evaluation through stratification, sampling, and estimation." In European Conference on Computer Vision, pp. 140-158. Cham: Springer Nature Switzerland, 2024.

Examples:

>>> import numpy as np
>>> from glide.monitors import AsymptoticStratifiedPPRM
>>> pre_drift_y_true = np.array([0.0, 0.2, np.nan, np.nan, 0.1, 0.3, np.nan, np.nan])
>>> pre_drift_y_proxy = np.array([0.0, 0.2, 0.0, 0.2, 0.1, 0.3, 0.1, 0.3])
>>> post_drift_y_true = np.array([0.8, 1.0, np.nan, np.nan, 0.7, 0.9, np.nan, np.nan])
>>> post_drift_y_proxy = np.array([0.8, 1.0, 0.8, 1.0, 0.7, 0.9, 0.7, 0.9])
>>> y_true = np.hstack([pre_drift_y_true, np.tile(post_drift_y_true, 5)])
>>> y_proxy = np.hstack([pre_drift_y_proxy, np.tile(post_drift_y_proxy, 5)])
>>> groups = np.tile([0, 0, 0, 0, 1, 1, 1, 1], 6)
>>> batches = np.repeat(np.arange(6), 8)
>>> monitor = AsymptoticStratifiedPPRM()
>>> result = monitor.detect(y_true, y_proxy, groups, batches, higher_is_better=False, threshold=0.5)
>>> result.drift_detected
True
>>> result.first_alarm_index
2
Source code in glide/monitors/asymptotic_stratified_pprm.py 
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
class AsymptoticStratifiedPPRM(AsymptoticRM[StratifiedPPIDataset, StratifiedTuningParameter]):
    """Anytime-valid drift monitor combining human and proxy labels over a stratified stream.

    Computes a per-batch Stratified PPI estimate together with its standard error.
    Each batch's samples are split into strata, a power-tuned PPI++ estimate is
    computed independently within each stratum using a weight fitted on the batches
    that strictly precede it, and the per-stratum estimates are combined with
    population-proportional weights. The monitor then tracks the running mean of the
    per-batch estimates the same way ``AsymptoticPPRM`` does for an unstratified
    stream. The false-alarm guarantee is asymptotic: each batch needs enough labeled
    and proxy samples, within each of its strata, for its Stratified PPI estimate to
    be approximately Gaussian with a consistently estimated variance.

    References
    ----------
    Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya
    Ramdas. "Time-uniform central limit theory and asymptotic confidence
    sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

    Zhang, Guangyi, Yunlong Cai, Guanding Yu, and Osvaldo Simeone. "Prediction-powered
    risk monitoring of deployed models for detecting harmful distribution shifts."
    arXiv preprint arXiv:2602.02229 (2026).

    Podkopaev, Aleksandr, and Aaditya Ramdas. "Tracking the risk of a deployed model
    and detecting harmful distribution shifts." International Conference on Learning
    Representations (ICLR), 2022.

    Fisch, Adam, Joshua Maynez, R. Alex Hofer, Bhuwan Dhingra, Amir Globerson, and
    William W. Cohen. "Stratified prediction-powered inference for effective hybrid
    evaluation of language models." Advances in Neural Information Processing
    Systems 37 (2024): 111489-111514.

    Fogliato, Riccardo, Pratik Patil, Mathew Monfort, and Pietro Perona. "A framework
    for efficient model evaluation through stratification, sampling, and estimation."
    In European Conference on Computer Vision, pp. 140-158. Cham: Springer Nature
    Switzerland, 2024.

    Examples
    --------
    >>> import numpy as np
    >>> from glide.monitors import AsymptoticStratifiedPPRM
    >>> pre_drift_y_true = np.array([0.0, 0.2, np.nan, np.nan, 0.1, 0.3, np.nan, np.nan])
    >>> pre_drift_y_proxy = np.array([0.0, 0.2, 0.0, 0.2, 0.1, 0.3, 0.1, 0.3])
    >>> post_drift_y_true = np.array([0.8, 1.0, np.nan, np.nan, 0.7, 0.9, np.nan, np.nan])
    >>> post_drift_y_proxy = np.array([0.8, 1.0, 0.8, 1.0, 0.7, 0.9, 0.7, 0.9])
    >>> y_true = np.hstack([pre_drift_y_true, np.tile(post_drift_y_true, 5)])
    >>> y_proxy = np.hstack([pre_drift_y_proxy, np.tile(post_drift_y_proxy, 5)])
    >>> groups = np.tile([0, 0, 0, 0, 1, 1, 1, 1], 6)
    >>> batches = np.repeat(np.arange(6), 8)
    >>> monitor = AsymptoticStratifiedPPRM()
    >>> result = monitor.detect(y_true, y_proxy, groups, batches, higher_is_better=False, threshold=0.5)
    >>> result.drift_detected
    True
    >>> result.first_alarm_index
    2
    """

    _engine = StratifiedPPIMeanEngine()

    def detect(
        self,
        y_true: NDArray,
        y_proxy: NDArray,
        groups: NDArray,
        batches: NDArray,
        *,
        higher_is_better: bool,
        threshold: float,
        metric_name: str = "Metric",
        confidence_level: float = 0.8,
        power_tuning: bool = True,
        tightest_at_batch: int = 10,
    ) -> PredictionPoweredMeanMonitoringResult:
        """Detect a drift of the running mean across a batched, stratified dataset.

        Splits the data by batch, then by stratum within each batch, computes a
        Stratified PPI estimate and its standard error per batch, and builds an
        anytime-valid asymptotic confidence sequence on the running mean of those
        estimates. An alarm is raised at every batch where the sequence crosses the
        user-supplied ``threshold``. See ``AsymptoticPPRM.detect`` for the full
        explanation of batch ordering, growing-history semantics, and
        ``tightest_at_batch``, which all carry over unchanged.

        Parameters
        ----------
        y_true : NDArray
            Array of labeled observations, shape ``(n_samples,)``.
            Labeled entries are finite; unlabeled entries are ``np.nan``.
        y_proxy : NDArray
            Array of proxy predictions, shape ``(n_samples,)``.
            Must be fully populated (no NaN).
        groups : NDArray
            Array of stratum identifiers, shape ``(n_samples,)``. Unique values
            within a batch define that batch's strata; a stratum need not appear in
            every batch. One appearing for the first time anywhere in the stream
            falls back to an unpowered estimate for that batch only (see ``power_tuning``).
        batches : NDArray
            Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
            oldest batch first and grouped into contiguous blocks. Identifier values
            are not compared, so any hashable label type works (integers, dates,
            free-form strings).
        higher_is_better : bool
            ``False`` when the metric is a risk (drift means the metric increased),
            ``True`` when it is a performance (drift means the metric decreased).
        threshold : float
            The metric value the running mean is monitored against, in metric units.
        metric_name : str, optional
            Human-readable label for the metric. Defaults to ``"Metric"``.
        confidence_level : float, optional
            How confident each alarm should be. Must be in ``(0.5, 1)``. Defaults to ``0.8``.
        power_tuning : bool, optional
            If ``True`` (default), compute each stratum's power-tuning parameter on
            all previous batches (a stratum's first appearance in the stream uses a
            fixed weight, having no predecessor to fit on). If ``False``, use a fixed
            weight everywhere.
        tightest_at_batch : int, optional
            The batch index (1-indexed) at which the confidence sequence is tuned to
            be tightest. Defaults to ``10``.

        Returns
        -------
        PredictionPoweredMeanMonitoringResult
            Per-batch estimates, running means, anytime-valid confidence bounds,
            alarm flags, and the alarm threshold, all in the original metric
            orientation. ``batch_n_true``/``batch_n_proxy`` aggregate across all of a
            batch's strata.

        Raises
        ------
        ValueError
            - If ``batches`` is empty.
            - If ``y_true``, ``y_proxy``, ``groups`` and ``batches`` have different lengths.
            - If ``batches`` or ``groups`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
            - If ``confidence_level`` is not in ``(0.5, 1)``.
            - If any proxy value is NaN.
            - If batches are interleaved rather than grouped into contiguous blocks.
            - If any stratum in any batch has fewer than 2 labeled or fewer than 2 unlabeled samples.
            - If a stratum's proxy values are constant across a prefix set of batches (with ``power_tuning=True``).
            - If ``tightest_at_batch`` is not a positive integer.
            - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
        """
        batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
            label_fields={"y_true": y_true, "y_proxy": y_proxy},
            identifier_fields={"groups": groups},
            batches=batches,
            higher_is_better=higher_is_better,
            confidence_level=confidence_level,
            tightest_at_batch=tightest_at_batch,
            power_tuning=power_tuning,
        )
        confidence_sequence = AsymptoticConfidenceSequence(
            running_mean_estimates=running_means, confidence_bounds=confidence_bounds
        )
        labeled_mask = ~np.isnan(y_true)
        batch_n_true = np.bincount(batch_codes[labeled_mask])
        batch_n_proxy = np.bincount(batch_codes)
        result = PredictionPoweredMeanMonitoringResult(
            metric_name=metric_name,
            monitor_name=self.__class__.__name__,
            higher_is_better=higher_is_better,
            alarm_threshold=threshold,
            confidence_level=confidence_level,
            batch_mean_estimates=batch_mean_estimates,
            confidence_sequence=confidence_sequence,
            batch_n_true=batch_n_true,
            batch_n_proxy=batch_n_proxy,
        )
        return result

detect 

detect(
    y_true,
    y_proxy,
    groups,
    batches,
    *,
    higher_is_better,
    threshold,
    metric_name="Metric",
    confidence_level=0.8,
    power_tuning=True,
    tightest_at_batch=10,
)

Detect a drift of the running mean across a batched, stratified dataset.

Splits the data by batch, then by stratum within each batch, computes a Stratified PPI estimate and its standard error per batch, and builds an anytime-valid asymptotic confidence sequence on the running mean of those estimates. An alarm is raised at every batch where the sequence crosses the user-supplied threshold. See AsymptoticPPRM.detect for the full explanation of batch ordering, growing-history semantics, and tightest_at_batch, which all carry over unchanged.

Parameters:

Name Type Description Default
y_true NDArray

Array of labeled observations, shape (n_samples,). Labeled entries are finite; unlabeled entries are np.nan.

 required
y_proxy NDArray

Array of proxy predictions, shape (n_samples,). Must be fully populated (no NaN).

 required
groups NDArray

Array of stratum identifiers, shape (n_samples,). Unique values within a batch define that batch's strata; a stratum need not appear in every batch. One appearing for the first time anywhere in the stream falls back to an unpowered estimate for that batch only (see power_tuning).

 required
batches NDArray

Array of batch identifiers, shape (n_samples,). Rows must be ordered oldest batch first and grouped into contiguous blocks. Identifier values are not compared, so any hashable label type works (integers, dates, free-form strings).

 required
higher_is_better bool

False when the metric is a risk (drift means the metric increased), True when it is a performance (drift means the metric decreased).

 required
threshold float

The metric value the running mean is monitored against, in metric units.

 required
metric_name str

Human-readable label for the metric. Defaults to "Metric".

 'Metric'
confidence_level float

How confident each alarm should be. Must be in (0.5, 1). Defaults to 0.8.

 0.8
power_tuning bool

If True (default), compute each stratum's power-tuning parameter on all previous batches (a stratum's first appearance in the stream uses a fixed weight, having no predecessor to fit on). If False, use a fixed weight everywhere.

 True
tightest_at_batch int

The batch index (1-indexed) at which the confidence sequence is tuned to be tightest. Defaults to 10.

 10

Returns:

Type Description
PredictionPoweredMeanMonitoringResult

Per-batch estimates, running means, anytime-valid confidence bounds, alarm flags, and the alarm threshold, all in the original metric orientation. batch_n_true/batch_n_proxy aggregate across all of a batch's strata.

Raises:

Type Description
ValueError
  • If batches is empty.
  • If y_true, y_proxy, groups and batches have different lengths.
  • If batches or groups contains NaN values (numeric dtype) or None values (non-numeric dtype).
  • If confidence_level is not in (0.5, 1).
  • If any proxy value is NaN.
  • If batches are interleaved rather than grouped into contiguous blocks.
  • If any stratum in any batch has fewer than 2 labeled or fewer than 2 unlabeled samples.
  • If a stratum's proxy values are constant across a prefix set of batches (with power_tuning=True).
  • If tightest_at_batch is not a positive integer.
  • If the accumulated variance of the batch estimates up to tightest_at_batch is zero.
Source code in glide/monitors/asymptotic_stratified_pprm.py 
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
def detect(
    self,
    y_true: NDArray,
    y_proxy: NDArray,
    groups: NDArray,
    batches: NDArray,
    *,
    higher_is_better: bool,
    threshold: float,
    metric_name: str = "Metric",
    confidence_level: float = 0.8,
    power_tuning: bool = True,
    tightest_at_batch: int = 10,
) -> PredictionPoweredMeanMonitoringResult:
    """Detect a drift of the running mean across a batched, stratified dataset.

    Splits the data by batch, then by stratum within each batch, computes a
    Stratified PPI estimate and its standard error per batch, and builds an
    anytime-valid asymptotic confidence sequence on the running mean of those
    estimates. An alarm is raised at every batch where the sequence crosses the
    user-supplied ``threshold``. See ``AsymptoticPPRM.detect`` for the full
    explanation of batch ordering, growing-history semantics, and
    ``tightest_at_batch``, which all carry over unchanged.

    Parameters
    ----------
    y_true : NDArray
        Array of labeled observations, shape ``(n_samples,)``.
        Labeled entries are finite; unlabeled entries are ``np.nan``.
    y_proxy : NDArray
        Array of proxy predictions, shape ``(n_samples,)``.
        Must be fully populated (no NaN).
    groups : NDArray
        Array of stratum identifiers, shape ``(n_samples,)``. Unique values
        within a batch define that batch's strata; a stratum need not appear in
        every batch. One appearing for the first time anywhere in the stream
        falls back to an unpowered estimate for that batch only (see ``power_tuning``).
    batches : NDArray
        Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
        oldest batch first and grouped into contiguous blocks. Identifier values
        are not compared, so any hashable label type works (integers, dates,
        free-form strings).
    higher_is_better : bool
        ``False`` when the metric is a risk (drift means the metric increased),
        ``True`` when it is a performance (drift means the metric decreased).
    threshold : float
        The metric value the running mean is monitored against, in metric units.
    metric_name : str, optional
        Human-readable label for the metric. Defaults to ``"Metric"``.
    confidence_level : float, optional
        How confident each alarm should be. Must be in ``(0.5, 1)``. Defaults to ``0.8``.
    power_tuning : bool, optional
        If ``True`` (default), compute each stratum's power-tuning parameter on
        all previous batches (a stratum's first appearance in the stream uses a
        fixed weight, having no predecessor to fit on). If ``False``, use a fixed
        weight everywhere.
    tightest_at_batch : int, optional
        The batch index (1-indexed) at which the confidence sequence is tuned to
        be tightest. Defaults to ``10``.

    Returns
    -------
    PredictionPoweredMeanMonitoringResult
        Per-batch estimates, running means, anytime-valid confidence bounds,
        alarm flags, and the alarm threshold, all in the original metric
        orientation. ``batch_n_true``/``batch_n_proxy`` aggregate across all of a
        batch's strata.

    Raises
    ------
    ValueError
        - If ``batches`` is empty.
        - If ``y_true``, ``y_proxy``, ``groups`` and ``batches`` have different lengths.
        - If ``batches`` or ``groups`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
        - If ``confidence_level`` is not in ``(0.5, 1)``.
        - If any proxy value is NaN.
        - If batches are interleaved rather than grouped into contiguous blocks.
        - If any stratum in any batch has fewer than 2 labeled or fewer than 2 unlabeled samples.
        - If a stratum's proxy values are constant across a prefix set of batches (with ``power_tuning=True``).
        - If ``tightest_at_batch`` is not a positive integer.
        - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
    """
    batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
        label_fields={"y_true": y_true, "y_proxy": y_proxy},
        identifier_fields={"groups": groups},
        batches=batches,
        higher_is_better=higher_is_better,
        confidence_level=confidence_level,
        tightest_at_batch=tightest_at_batch,
        power_tuning=power_tuning,
    )
    confidence_sequence = AsymptoticConfidenceSequence(
        running_mean_estimates=running_means, confidence_bounds=confidence_bounds
    )
    labeled_mask = ~np.isnan(y_true)
    batch_n_true = np.bincount(batch_codes[labeled_mask])
    batch_n_proxy = np.bincount(batch_codes)
    result = PredictionPoweredMeanMonitoringResult(
        metric_name=metric_name,
        monitor_name=self.__class__.__name__,
        higher_is_better=higher_is_better,
        alarm_threshold=threshold,
        confidence_level=confidence_level,
        batch_mean_estimates=batch_mean_estimates,
        confidence_sequence=confidence_sequence,
        batch_n_true=batch_n_true,
        batch_n_proxy=batch_n_proxy,
    )
    return result

glide.monitors.asymptotic_clustered_pprm.AsymptoticClusteredPPRM

Bases: AsymptoticRM[ClusteredPPIDataset, float]

Anytime-valid drift monitor combining human and proxy labels over clustered data.

Computes a per-batch Clustered PPI estimate (each batch's observations are reduced to per-cluster means, and a power-tuned PPI++ estimate is computed on those means using a weight fitted on the batches that strictly precede it, predictable as required for validity), together with its standard error, and tracks the running mean of the per-batch estimates the same way AsymptoticPPRM does for independent, unclustered samples.

References

Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya Ramdas. "Time-uniform central limit theory and asymptotic confidence sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

Zhang, Guangyi, Yunlong Cai, Guanding Yu, and Osvaldo Simeone. "Prediction-powered risk monitoring of deployed models for detecting harmful distribution shifts." arXiv preprint arXiv:2602.02229 (2026).

Podkopaev, Aleksandr, and Aaditya Ramdas. "Tracking the risk of a deployed model and detecting harmful distribution shifts." International Conference on Learning Representations (ICLR), 2022.

Broska, David. "Cluster-robust PPI reference implementation." https://github.com/davidbroska/ppi_py/blob/main/ClusterPPI/mean.py

Examples:

>>> import numpy as np
>>> from glide.monitors import AsymptoticClusteredPPRM
>>> pre_drift_y_true = np.array([0.0, 0.2, np.nan, np.nan, 0.1, 0.3, np.nan, np.nan])
>>> pre_drift_y_proxy = np.array([0.0, 0.2, 0.0, 0.2, 0.1, 0.3, 0.1, 0.3])
>>> post_drift_y_true = np.array([0.8, 1.0, np.nan, np.nan, 0.7, 0.9, np.nan, np.nan])
>>> post_drift_y_proxy = np.array([0.8, 1.0, 0.8, 1.0, 0.7, 0.9, 0.7, 0.9])
>>> y_true = np.hstack([pre_drift_y_true, np.tile(post_drift_y_true, 5)])
>>> y_proxy = np.hstack([pre_drift_y_proxy, np.tile(post_drift_y_proxy, 5)])
>>> clusters = np.repeat(np.arange(24), 2)
>>> batches = np.repeat(np.arange(6), 8)
>>> monitor = AsymptoticClusteredPPRM()
>>> result = monitor.detect(y_true, y_proxy, clusters, batches, higher_is_better=False, threshold=0.5)
>>> result.drift_detected
True
>>> result.first_alarm_index
2
Source code in glide/monitors/asymptotic_clustered_pprm.py 
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
class AsymptoticClusteredPPRM(AsymptoticRM[ClusteredPPIDataset, float]):
    """Anytime-valid drift monitor combining human and proxy labels over clustered data.

    Computes a per-batch Clustered PPI estimate (each batch's observations are reduced
    to per-cluster means, and a power-tuned PPI++ estimate is computed on those means
    using a weight fitted on the batches that strictly precede it, predictable as
    required for validity), together with its standard error, and tracks the running
    mean of the per-batch estimates the same way ``AsymptoticPPRM`` does for
    independent, unclustered samples.

    References
    ----------
    Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya
    Ramdas. "Time-uniform central limit theory and asymptotic confidence
    sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

    Zhang, Guangyi, Yunlong Cai, Guanding Yu, and Osvaldo Simeone. "Prediction-powered
    risk monitoring of deployed models for detecting harmful distribution shifts."
    arXiv preprint arXiv:2602.02229 (2026).

    Podkopaev, Aleksandr, and Aaditya Ramdas. "Tracking the risk of a deployed model
    and detecting harmful distribution shifts." International Conference on Learning
    Representations (ICLR), 2022.

    Broska, David. "Cluster-robust PPI reference implementation."
    https://github.com/davidbroska/ppi_py/blob/main/ClusterPPI/mean.py

    Examples
    --------
    >>> import numpy as np
    >>> from glide.monitors import AsymptoticClusteredPPRM
    >>> pre_drift_y_true = np.array([0.0, 0.2, np.nan, np.nan, 0.1, 0.3, np.nan, np.nan])
    >>> pre_drift_y_proxy = np.array([0.0, 0.2, 0.0, 0.2, 0.1, 0.3, 0.1, 0.3])
    >>> post_drift_y_true = np.array([0.8, 1.0, np.nan, np.nan, 0.7, 0.9, np.nan, np.nan])
    >>> post_drift_y_proxy = np.array([0.8, 1.0, 0.8, 1.0, 0.7, 0.9, 0.7, 0.9])
    >>> y_true = np.hstack([pre_drift_y_true, np.tile(post_drift_y_true, 5)])
    >>> y_proxy = np.hstack([pre_drift_y_proxy, np.tile(post_drift_y_proxy, 5)])
    >>> clusters = np.repeat(np.arange(24), 2)
    >>> batches = np.repeat(np.arange(6), 8)
    >>> monitor = AsymptoticClusteredPPRM()
    >>> result = monitor.detect(y_true, y_proxy, clusters, batches, higher_is_better=False, threshold=0.5)
    >>> result.drift_detected
    True
    >>> result.first_alarm_index
    2
    """

    _engine = ClusteredPPIMeanEngine()

    def detect(
        self,
        y_true: NDArray,
        y_proxy: NDArray,
        clusters: NDArray,
        batches: NDArray,
        *,
        higher_is_better: bool,
        threshold: float,
        metric_name: str = "Metric",
        confidence_level: float = 0.8,
        power_tuning: bool = True,
        tightest_at_batch: int = 10,
    ) -> PredictionPoweredMeanMonitoringResult:
        """Detect a drift of the running mean across a batched, clustered dataset.

        Splits the data by batch, reduces each batch's observations to per-cluster
        means, computes a Clustered PPI estimate and its standard error per batch, and
        builds an anytime-valid asymptotic confidence sequence on the running mean of
        those estimates. An alarm is raised at every batch where the sequence crosses
        the user-supplied ``threshold``. See ``AsymptoticPPRM.detect`` for the full
        explanation of batch ordering, growing-history semantics, and
        ``tightest_at_batch``, which all carry over unchanged.

        Parameters
        ----------
        y_true : NDArray
            Array of labeled observations, shape ``(n_samples,)``.
            Labeled entries are finite; unlabeled entries are ``np.nan``. All
            observations in the same cluster must share the same label status.
        y_proxy : NDArray
            Array of proxy predictions, shape ``(n_samples,)``.
            Must be fully populated (no NaN).
        clusters : NDArray
            Array of cluster identifiers, shape ``(n_samples,)``. Any hashable label
            type works (integers, strings, ...). Must be unique across the whole
            stream: an identifier reused in two different batches raises, since
            their rows would otherwise be silently pooled into a single artificial
            cluster once both fall inside the same prefix.
        batches : NDArray
            Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
            oldest batch first and grouped into contiguous blocks. Identifier values
            are not compared, so any hashable label type works (integers, dates,
            free-form strings).
        higher_is_better : bool
            ``False`` when the metric is a risk (drift means the metric increased),
            ``True`` when it is a performance (drift means the metric decreased).
        threshold : float
            The metric value the running mean is monitored against, in metric units.
        metric_name : str, optional
            Human-readable label for the metric. Defaults to ``"Metric"``.
        confidence_level : float, optional
            How confident each alarm should be. Must be in ``(0.5, 1)``. Defaults to ``0.8``.
        power_tuning : bool, optional
            If ``True`` (default), compute the power-tuning parameter from the cluster
            means of all previous batches, pooled together (the first batch, having no
            predecessor, uses a fixed weight). If ``False``, use a fixed weight everywhere.
        tightest_at_batch : int, optional
            The batch index (1-indexed) at which the confidence sequence is tuned to
            be tightest. Defaults to ``10``.

        Returns
        -------
        PredictionPoweredMeanMonitoringResult
            Per-batch estimates, running means, anytime-valid confidence bounds,
            alarm flags, and the alarm threshold, all in the original metric
            orientation. ``batch_n_true``/``batch_n_proxy`` aggregate across all of a
            batch's clusters.

        Raises
        ------
        ValueError
            - If ``batches`` is empty.
            - If ``y_true``, ``y_proxy``, ``clusters`` and ``batches`` have different lengths.
            - If ``batches`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
            - If ``clusters`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
            - If ``confidence_level`` is not in ``(0.5, 1)``.
            - If any proxy value is NaN.
            - If batches are interleaved rather than grouped into contiguous blocks.
            - If any batch has a cluster containing both labeled and unlabeled observations.
            - If any batch has fewer than 2 fully labeled or fewer than 2 fully unlabeled clusters.
            - If a cluster identifier's rows span more than one batch.
            - If proxy cluster means are constant across a prefix set of batches (with ``power_tuning=True``).
            - If ``tightest_at_batch`` is not a positive integer.
            - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
        """
        _validate_clusters_unique_across_batches(clusters, batches)
        batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
            label_fields={"y_true": y_true, "y_proxy": y_proxy},
            identifier_fields={"clusters": clusters},
            batches=batches,
            higher_is_better=higher_is_better,
            confidence_level=confidence_level,
            tightest_at_batch=tightest_at_batch,
            power_tuning=power_tuning,
        )
        confidence_sequence = AsymptoticConfidenceSequence(
            running_mean_estimates=running_means, confidence_bounds=confidence_bounds
        )
        labeled_mask = ~np.isnan(y_true)
        batch_n_true = np.bincount(batch_codes[labeled_mask])
        batch_n_proxy = np.bincount(batch_codes)
        result = PredictionPoweredMeanMonitoringResult(
            metric_name=metric_name,
            monitor_name=self.__class__.__name__,
            higher_is_better=higher_is_better,
            alarm_threshold=threshold,
            confidence_level=confidence_level,
            batch_mean_estimates=batch_mean_estimates,
            confidence_sequence=confidence_sequence,
            batch_n_true=batch_n_true,
            batch_n_proxy=batch_n_proxy,
        )
        return result

detect 

detect(
    y_true,
    y_proxy,
    clusters,
    batches,
    *,
    higher_is_better,
    threshold,
    metric_name="Metric",
    confidence_level=0.8,
    power_tuning=True,
    tightest_at_batch=10,
)

Detect a drift of the running mean across a batched, clustered dataset.

Splits the data by batch, reduces each batch's observations to per-cluster means, computes a Clustered PPI estimate and its standard error per batch, and builds an anytime-valid asymptotic confidence sequence on the running mean of those estimates. An alarm is raised at every batch where the sequence crosses the user-supplied threshold. See AsymptoticPPRM.detect for the full explanation of batch ordering, growing-history semantics, and tightest_at_batch, which all carry over unchanged.

Parameters:

Name Type Description Default
y_true NDArray

Array of labeled observations, shape (n_samples,). Labeled entries are finite; unlabeled entries are np.nan. All observations in the same cluster must share the same label status.

 required
y_proxy NDArray

Array of proxy predictions, shape (n_samples,). Must be fully populated (no NaN).

 required
clusters NDArray

Array of cluster identifiers, shape (n_samples,). Any hashable label type works (integers, strings, ...). Must be unique across the whole stream: an identifier reused in two different batches raises, since their rows would otherwise be silently pooled into a single artificial cluster once both fall inside the same prefix.

 required
batches NDArray

Array of batch identifiers, shape (n_samples,). Rows must be ordered oldest batch first and grouped into contiguous blocks. Identifier values are not compared, so any hashable label type works (integers, dates, free-form strings).

 required
higher_is_better bool

False when the metric is a risk (drift means the metric increased), True when it is a performance (drift means the metric decreased).

 required
threshold float

The metric value the running mean is monitored against, in metric units.

 required
metric_name str

Human-readable label for the metric. Defaults to "Metric".

 'Metric'
confidence_level float

How confident each alarm should be. Must be in (0.5, 1). Defaults to 0.8.

 0.8
power_tuning bool

If True (default), compute the power-tuning parameter from the cluster means of all previous batches, pooled together (the first batch, having no predecessor, uses a fixed weight). If False, use a fixed weight everywhere.

 True
tightest_at_batch int

The batch index (1-indexed) at which the confidence sequence is tuned to be tightest. Defaults to 10.

 10

Returns:

Type Description
PredictionPoweredMeanMonitoringResult

Per-batch estimates, running means, anytime-valid confidence bounds, alarm flags, and the alarm threshold, all in the original metric orientation. batch_n_true/batch_n_proxy aggregate across all of a batch's clusters.

Raises:

Type Description
ValueError
  • If batches is empty.
  • If y_true, y_proxy, clusters and batches have different lengths.
  • If batches contains NaN values (numeric dtype) or None values (non-numeric dtype).
  • If clusters contains NaN values (numeric dtype) or None values (non-numeric dtype).
  • If confidence_level is not in (0.5, 1).
  • If any proxy value is NaN.
  • If batches are interleaved rather than grouped into contiguous blocks.
  • If any batch has a cluster containing both labeled and unlabeled observations.
  • If any batch has fewer than 2 fully labeled or fewer than 2 fully unlabeled clusters.
  • If a cluster identifier's rows span more than one batch.
  • If proxy cluster means are constant across a prefix set of batches (with power_tuning=True).
  • If tightest_at_batch is not a positive integer.
  • If the accumulated variance of the batch estimates up to tightest_at_batch is zero.
Source code in glide/monitors/asymptotic_clustered_pprm.py 
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
def detect(
    self,
    y_true: NDArray,
    y_proxy: NDArray,
    clusters: NDArray,
    batches: NDArray,
    *,
    higher_is_better: bool,
    threshold: float,
    metric_name: str = "Metric",
    confidence_level: float = 0.8,
    power_tuning: bool = True,
    tightest_at_batch: int = 10,
) -> PredictionPoweredMeanMonitoringResult:
    """Detect a drift of the running mean across a batched, clustered dataset.

    Splits the data by batch, reduces each batch's observations to per-cluster
    means, computes a Clustered PPI estimate and its standard error per batch, and
    builds an anytime-valid asymptotic confidence sequence on the running mean of
    those estimates. An alarm is raised at every batch where the sequence crosses
    the user-supplied ``threshold``. See ``AsymptoticPPRM.detect`` for the full
    explanation of batch ordering, growing-history semantics, and
    ``tightest_at_batch``, which all carry over unchanged.

    Parameters
    ----------
    y_true : NDArray
        Array of labeled observations, shape ``(n_samples,)``.
        Labeled entries are finite; unlabeled entries are ``np.nan``. All
        observations in the same cluster must share the same label status.
    y_proxy : NDArray
        Array of proxy predictions, shape ``(n_samples,)``.
        Must be fully populated (no NaN).
    clusters : NDArray
        Array of cluster identifiers, shape ``(n_samples,)``. Any hashable label
        type works (integers, strings, ...). Must be unique across the whole
        stream: an identifier reused in two different batches raises, since
        their rows would otherwise be silently pooled into a single artificial
        cluster once both fall inside the same prefix.
    batches : NDArray
        Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
        oldest batch first and grouped into contiguous blocks. Identifier values
        are not compared, so any hashable label type works (integers, dates,
        free-form strings).
    higher_is_better : bool
        ``False`` when the metric is a risk (drift means the metric increased),
        ``True`` when it is a performance (drift means the metric decreased).
    threshold : float
        The metric value the running mean is monitored against, in metric units.
    metric_name : str, optional
        Human-readable label for the metric. Defaults to ``"Metric"``.
    confidence_level : float, optional
        How confident each alarm should be. Must be in ``(0.5, 1)``. Defaults to ``0.8``.
    power_tuning : bool, optional
        If ``True`` (default), compute the power-tuning parameter from the cluster
        means of all previous batches, pooled together (the first batch, having no
        predecessor, uses a fixed weight). If ``False``, use a fixed weight everywhere.
    tightest_at_batch : int, optional
        The batch index (1-indexed) at which the confidence sequence is tuned to
        be tightest. Defaults to ``10``.

    Returns
    -------
    PredictionPoweredMeanMonitoringResult
        Per-batch estimates, running means, anytime-valid confidence bounds,
        alarm flags, and the alarm threshold, all in the original metric
        orientation. ``batch_n_true``/``batch_n_proxy`` aggregate across all of a
        batch's clusters.

    Raises
    ------
    ValueError
        - If ``batches`` is empty.
        - If ``y_true``, ``y_proxy``, ``clusters`` and ``batches`` have different lengths.
        - If ``batches`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
        - If ``clusters`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
        - If ``confidence_level`` is not in ``(0.5, 1)``.
        - If any proxy value is NaN.
        - If batches are interleaved rather than grouped into contiguous blocks.
        - If any batch has a cluster containing both labeled and unlabeled observations.
        - If any batch has fewer than 2 fully labeled or fewer than 2 fully unlabeled clusters.
        - If a cluster identifier's rows span more than one batch.
        - If proxy cluster means are constant across a prefix set of batches (with ``power_tuning=True``).
        - If ``tightest_at_batch`` is not a positive integer.
        - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
    """
    _validate_clusters_unique_across_batches(clusters, batches)
    batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
        label_fields={"y_true": y_true, "y_proxy": y_proxy},
        identifier_fields={"clusters": clusters},
        batches=batches,
        higher_is_better=higher_is_better,
        confidence_level=confidence_level,
        tightest_at_batch=tightest_at_batch,
        power_tuning=power_tuning,
    )
    confidence_sequence = AsymptoticConfidenceSequence(
        running_mean_estimates=running_means, confidence_bounds=confidence_bounds
    )
    labeled_mask = ~np.isnan(y_true)
    batch_n_true = np.bincount(batch_codes[labeled_mask])
    batch_n_proxy = np.bincount(batch_codes)
    result = PredictionPoweredMeanMonitoringResult(
        metric_name=metric_name,
        monitor_name=self.__class__.__name__,
        higher_is_better=higher_is_better,
        alarm_threshold=threshold,
        confidence_level=confidence_level,
        batch_mean_estimates=batch_mean_estimates,
        confidence_sequence=confidence_sequence,
        batch_n_true=batch_n_true,
        batch_n_proxy=batch_n_proxy,
    )
    return result

glide.monitors.asymptotic_multi_pprm.AsymptoticMultiPPRM

Bases: AsymptoticRM[MultiPPIDataset, NDArray]

Anytime-valid drift monitor combining human labels with multiple proxy predictors.

Computes a per-batch Multi-PPI estimate (a small set of true labels and several sets of proxy labels, combined through a power-tuning weight vector fitted on the batches that strictly precede it), together with its standard error, and tracks the running mean of the per-batch estimates the same way AsymptoticPPRM does for a single proxy. The false-alarm guarantee is asymptotic: each batch needs enough labeled and proxy samples for its Multi-PPI estimate to be approximately Gaussian with a consistently estimated variance.

References

Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya Ramdas. "Time-uniform central limit theory and asymptotic confidence sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

Zhang, Guangyi, Yunlong Cai, Guanding Yu, and Osvaldo Simeone. "Prediction-powered risk monitoring of deployed models for detecting harmful distribution shifts." arXiv preprint arXiv:2602.02229 (2026).

Podkopaev, Aleksandr, and Aaditya Ramdas. "Tracking the risk of a deployed model and detecting harmful distribution shifts." International Conference on Learning Representations (ICLR), 2022.

Shan, Jiawei, Zhifeng Chen, Yiming Dong, Yazhen Wang, and Jiwei Zhao. "SADA: Safe and Adaptive Aggregation of Multiple Black-Box Predictions in Semi-Supervised Learning." arXiv preprint arXiv:2509.21707 (2025).

Examples:

>>> import numpy as np
>>> from glide.monitors import AsymptoticMultiPPRM
>>> pre_drift_y_true = np.array([0.0, 0.2, np.nan, np.nan])
>>> pre_drift_y_proxies = np.array([[0.0, 0.1], [0.2, 0.3], [0.05, 0.25], [0.15, 0.05]])
>>> post_drift_y_true = np.array([0.8, 1.0, np.nan, np.nan])
>>> post_drift_y_proxies = pre_drift_y_proxies + 0.8
>>> y_true = np.hstack([pre_drift_y_true, np.tile(post_drift_y_true, 5)])
>>> y_proxies = np.vstack([pre_drift_y_proxies, np.tile(post_drift_y_proxies, (5, 1))])
>>> batches = np.repeat(np.arange(6), 4)
>>> monitor = AsymptoticMultiPPRM()
>>> result = monitor.detect(y_true, y_proxies, batches, higher_is_better=False, threshold=0.5)
>>> result.drift_detected
True
>>> result.first_alarm_index
2
Source code in glide/monitors/asymptotic_multi_pprm.py 
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
class AsymptoticMultiPPRM(AsymptoticRM[MultiPPIDataset, NDArray]):
    """Anytime-valid drift monitor combining human labels with multiple proxy predictors.

    Computes a per-batch Multi-PPI estimate (a small set of true labels and several
    sets of proxy labels, combined through a power-tuning weight vector fitted on the
    batches that strictly precede it), together with its standard error, and tracks
    the running mean of the per-batch estimates the same way ``AsymptoticPPRM`` does
    for a single proxy. The false-alarm guarantee is asymptotic: each batch needs
    enough labeled and proxy samples for its Multi-PPI estimate to be approximately
    Gaussian with a consistently estimated variance.

    References
    ----------
    Waudby-Smith, Ian, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya
    Ramdas. "Time-uniform central limit theory and asymptotic confidence
    sequences." The Annals of Statistics 52, no. 6 (2024): 2613-2640.

    Zhang, Guangyi, Yunlong Cai, Guanding Yu, and Osvaldo Simeone. "Prediction-powered
    risk monitoring of deployed models for detecting harmful distribution shifts."
    arXiv preprint arXiv:2602.02229 (2026).

    Podkopaev, Aleksandr, and Aaditya Ramdas. "Tracking the risk of a deployed model
    and detecting harmful distribution shifts." International Conference on Learning
    Representations (ICLR), 2022.

    Shan, Jiawei, Zhifeng Chen, Yiming Dong, Yazhen Wang, and Jiwei Zhao. "SADA: Safe
    and Adaptive Aggregation of Multiple Black-Box Predictions in Semi-Supervised
    Learning." arXiv preprint arXiv:2509.21707 (2025).

    Examples
    --------
    >>> import numpy as np
    >>> from glide.monitors import AsymptoticMultiPPRM
    >>> pre_drift_y_true = np.array([0.0, 0.2, np.nan, np.nan])
    >>> pre_drift_y_proxies = np.array([[0.0, 0.1], [0.2, 0.3], [0.05, 0.25], [0.15, 0.05]])
    >>> post_drift_y_true = np.array([0.8, 1.0, np.nan, np.nan])
    >>> post_drift_y_proxies = pre_drift_y_proxies + 0.8
    >>> y_true = np.hstack([pre_drift_y_true, np.tile(post_drift_y_true, 5)])
    >>> y_proxies = np.vstack([pre_drift_y_proxies, np.tile(post_drift_y_proxies, (5, 1))])
    >>> batches = np.repeat(np.arange(6), 4)
    >>> monitor = AsymptoticMultiPPRM()
    >>> result = monitor.detect(y_true, y_proxies, batches, higher_is_better=False, threshold=0.5)
    >>> result.drift_detected
    True
    >>> result.first_alarm_index
    2
    """

    _engine = MultiPPIMeanEngine()

    def detect(
        self,
        y_true: NDArray,
        y_proxies: NDArray,
        batches: NDArray,
        *,
        higher_is_better: bool,
        threshold: float,
        metric_name: str = "Metric",
        confidence_level: float = 0.8,
        power_tuning: bool = True,
        tightest_at_batch: int = 10,
    ) -> PredictionPoweredMeanMonitoringResult:
        """Detect a drift of the running mean across a batched dataset.

        Splits the data by batch, computes a Multi-PPI estimate and its standard error
        per batch, and builds an anytime-valid asymptotic confidence sequence on the
        running mean of those estimates. An alarm is raised at every batch where the
        sequence crosses the user-supplied ``threshold``. See ``AsymptoticPPRM.detect``
        for the full explanation of batch ordering, growing-history semantics, and
        ``tightest_at_batch``, which all carry over unchanged.

        Parameters
        ----------
        y_true : NDArray
            Array of labeled observations, shape ``(n_samples,)``.
            Labeled entries are finite; unlabeled entries are ``np.nan``.
        y_proxies : NDArray
            2D array of proxy predictions, shape ``(n_samples, n_proxies)``.
            Must be fully populated (no NaN). Each column must have nonzero variance
            when ``power_tuning=True``.
        batches : NDArray
            Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
            oldest batch first and grouped into contiguous blocks. Identifier values
            are not compared, so any hashable label type works (integers, dates,
            free-form strings).
        higher_is_better : bool
            ``False`` when the metric is a risk (drift means the metric increased),
            ``True`` when it is a performance (drift means the metric decreased).
        threshold : float
            The metric value the running mean is monitored against, in metric units.
        metric_name : str, optional
            Human-readable label for the metric. Defaults to ``"Metric"``.
        confidence_level : float, optional
            How confident each alarm should be. Must be in ``(0.5, 1)``. Defaults to ``0.8``.
        power_tuning : bool, optional
            If ``True`` (default), compute the power-tuning parameter vector of each
            batch on all previous batches (the first batch, having no predecessor,
            uses a fixed weight). If ``False``, use a fixed weight everywhere.
        tightest_at_batch : int, optional
            The batch index (1-indexed) at which the confidence sequence is tuned to
            be tightest. Defaults to ``10``.

        Returns
        -------
        PredictionPoweredMeanMonitoringResult
            Per-batch estimates, running means, anytime-valid confidence bounds,
            alarm flags, and the alarm threshold, all in the original metric
            orientation. ``batch_n_proxy`` counts the shared row count of
            ``y_proxies``, unaffected by the number of proxy columns.

        Raises
        ------
        ValueError
            - If ``batches`` is empty.
            - If ``y_true``, ``y_proxies`` and ``batches`` have different lengths.
            - If ``y_proxies`` is not a 2D array.
            - If ``batches`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
            - If ``confidence_level`` is not in ``(0.5, 1)``.
            - If any proxy value is NaN.
            - If batches are interleaved rather than grouped into contiguous blocks.
            - If any batch has fewer than 2 labeled or fewer than 2 unlabeled samples.
            - If any proxy column is constant across a prefix set of batches (with ``power_tuning=True``).
            - If the proxy covariance matrix is singular (e.g. two or more columns are
              perfectly correlated) for a prefix set of batches, with ``power_tuning=True``.
            - If ``tightest_at_batch`` is not a positive integer.
            - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
        """
        batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
            label_fields={"y_true": y_true, "y_proxies": y_proxies},
            batches=batches,
            higher_is_better=higher_is_better,
            confidence_level=confidence_level,
            tightest_at_batch=tightest_at_batch,
            power_tuning=power_tuning,
        )
        confidence_sequence = AsymptoticConfidenceSequence(
            running_mean_estimates=running_means, confidence_bounds=confidence_bounds
        )
        labeled_mask = ~np.isnan(y_true)
        batch_n_true = np.bincount(batch_codes[labeled_mask])
        batch_n_proxy = np.bincount(batch_codes)
        result = PredictionPoweredMeanMonitoringResult(
            metric_name=metric_name,
            monitor_name=self.__class__.__name__,
            higher_is_better=higher_is_better,
            alarm_threshold=threshold,
            confidence_level=confidence_level,
            batch_mean_estimates=batch_mean_estimates,
            confidence_sequence=confidence_sequence,
            batch_n_true=batch_n_true,
            batch_n_proxy=batch_n_proxy,
        )
        return result

detect 

detect(
    y_true,
    y_proxies,
    batches,
    *,
    higher_is_better,
    threshold,
    metric_name="Metric",
    confidence_level=0.8,
    power_tuning=True,
    tightest_at_batch=10,
)

Detect a drift of the running mean across a batched dataset.

Splits the data by batch, computes a Multi-PPI estimate and its standard error per batch, and builds an anytime-valid asymptotic confidence sequence on the running mean of those estimates. An alarm is raised at every batch where the sequence crosses the user-supplied threshold. See AsymptoticPPRM.detect for the full explanation of batch ordering, growing-history semantics, and tightest_at_batch, which all carry over unchanged.

Parameters:

Name Type Description Default
y_true NDArray

Array of labeled observations, shape (n_samples,). Labeled entries are finite; unlabeled entries are np.nan.

 required
y_proxies NDArray

2D array of proxy predictions, shape (n_samples, n_proxies). Must be fully populated (no NaN). Each column must have nonzero variance when power_tuning=True.

 required
batches NDArray

Array of batch identifiers, shape (n_samples,). Rows must be ordered oldest batch first and grouped into contiguous blocks. Identifier values are not compared, so any hashable label type works (integers, dates, free-form strings).

 required
higher_is_better bool

False when the metric is a risk (drift means the metric increased), True when it is a performance (drift means the metric decreased).

 required
threshold float

The metric value the running mean is monitored against, in metric units.

 required
metric_name str

Human-readable label for the metric. Defaults to "Metric".

 'Metric'
confidence_level float

How confident each alarm should be. Must be in (0.5, 1). Defaults to 0.8.

 0.8
power_tuning bool

If True (default), compute the power-tuning parameter vector of each batch on all previous batches (the first batch, having no predecessor, uses a fixed weight). If False, use a fixed weight everywhere.

 True
tightest_at_batch int

The batch index (1-indexed) at which the confidence sequence is tuned to be tightest. Defaults to 10.

 10

Returns:

Type Description
PredictionPoweredMeanMonitoringResult

Per-batch estimates, running means, anytime-valid confidence bounds, alarm flags, and the alarm threshold, all in the original metric orientation. batch_n_proxy counts the shared row count of y_proxies, unaffected by the number of proxy columns.

Raises:

Type Description
ValueError
  • If batches is empty.
  • If y_true, y_proxies and batches have different lengths.
  • If y_proxies is not a 2D array.
  • If batches contains NaN values (numeric dtype) or None values (non-numeric dtype).
  • If confidence_level is not in (0.5, 1).
  • If any proxy value is NaN.
  • If batches are interleaved rather than grouped into contiguous blocks.
  • If any batch has fewer than 2 labeled or fewer than 2 unlabeled samples.
  • If any proxy column is constant across a prefix set of batches (with power_tuning=True).
  • If the proxy covariance matrix is singular (e.g. two or more columns are perfectly correlated) for a prefix set of batches, with power_tuning=True.
  • If tightest_at_batch is not a positive integer.
  • If the accumulated variance of the batch estimates up to tightest_at_batch is zero.
Source code in glide/monitors/asymptotic_multi_pprm.py 
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
def detect(
    self,
    y_true: NDArray,
    y_proxies: NDArray,
    batches: NDArray,
    *,
    higher_is_better: bool,
    threshold: float,
    metric_name: str = "Metric",
    confidence_level: float = 0.8,
    power_tuning: bool = True,
    tightest_at_batch: int = 10,
) -> PredictionPoweredMeanMonitoringResult:
    """Detect a drift of the running mean across a batched dataset.

    Splits the data by batch, computes a Multi-PPI estimate and its standard error
    per batch, and builds an anytime-valid asymptotic confidence sequence on the
    running mean of those estimates. An alarm is raised at every batch where the
    sequence crosses the user-supplied ``threshold``. See ``AsymptoticPPRM.detect``
    for the full explanation of batch ordering, growing-history semantics, and
    ``tightest_at_batch``, which all carry over unchanged.

    Parameters
    ----------
    y_true : NDArray
        Array of labeled observations, shape ``(n_samples,)``.
        Labeled entries are finite; unlabeled entries are ``np.nan``.
    y_proxies : NDArray
        2D array of proxy predictions, shape ``(n_samples, n_proxies)``.
        Must be fully populated (no NaN). Each column must have nonzero variance
        when ``power_tuning=True``.
    batches : NDArray
        Array of batch identifiers, shape ``(n_samples,)``. Rows must be ordered
        oldest batch first and grouped into contiguous blocks. Identifier values
        are not compared, so any hashable label type works (integers, dates,
        free-form strings).
    higher_is_better : bool
        ``False`` when the metric is a risk (drift means the metric increased),
        ``True`` when it is a performance (drift means the metric decreased).
    threshold : float
        The metric value the running mean is monitored against, in metric units.
    metric_name : str, optional
        Human-readable label for the metric. Defaults to ``"Metric"``.
    confidence_level : float, optional
        How confident each alarm should be. Must be in ``(0.5, 1)``. Defaults to ``0.8``.
    power_tuning : bool, optional
        If ``True`` (default), compute the power-tuning parameter vector of each
        batch on all previous batches (the first batch, having no predecessor,
        uses a fixed weight). If ``False``, use a fixed weight everywhere.
    tightest_at_batch : int, optional
        The batch index (1-indexed) at which the confidence sequence is tuned to
        be tightest. Defaults to ``10``.

    Returns
    -------
    PredictionPoweredMeanMonitoringResult
        Per-batch estimates, running means, anytime-valid confidence bounds,
        alarm flags, and the alarm threshold, all in the original metric
        orientation. ``batch_n_proxy`` counts the shared row count of
        ``y_proxies``, unaffected by the number of proxy columns.

    Raises
    ------
    ValueError
        - If ``batches`` is empty.
        - If ``y_true``, ``y_proxies`` and ``batches`` have different lengths.
        - If ``y_proxies`` is not a 2D array.
        - If ``batches`` contains NaN values (numeric dtype) or None values (non-numeric dtype).
        - If ``confidence_level`` is not in ``(0.5, 1)``.
        - If any proxy value is NaN.
        - If batches are interleaved rather than grouped into contiguous blocks.
        - If any batch has fewer than 2 labeled or fewer than 2 unlabeled samples.
        - If any proxy column is constant across a prefix set of batches (with ``power_tuning=True``).
        - If the proxy covariance matrix is singular (e.g. two or more columns are
          perfectly correlated) for a prefix set of batches, with ``power_tuning=True``.
        - If ``tightest_at_batch`` is not a positive integer.
        - If the accumulated variance of the batch estimates up to ``tightest_at_batch`` is zero.
    """
    batch_codes, batch_mean_estimates, running_means, confidence_bounds = self._detect(
        label_fields={"y_true": y_true, "y_proxies": y_proxies},
        batches=batches,
        higher_is_better=higher_is_better,
        confidence_level=confidence_level,
        tightest_at_batch=tightest_at_batch,
        power_tuning=power_tuning,
    )
    confidence_sequence = AsymptoticConfidenceSequence(
        running_mean_estimates=running_means, confidence_bounds=confidence_bounds
    )
    labeled_mask = ~np.isnan(y_true)
    batch_n_true = np.bincount(batch_codes[labeled_mask])
    batch_n_proxy = np.bincount(batch_codes)
    result = PredictionPoweredMeanMonitoringResult(
        metric_name=metric_name,
        monitor_name=self.__class__.__name__,
        higher_is_better=higher_is_better,
        alarm_threshold=threshold,
        confidence_level=confidence_level,
        batch_mean_estimates=batch_mean_estimates,
        confidence_sequence=confidence_sequence,
        batch_n_true=batch_n_true,
        batch_n_proxy=batch_n_proxy,
    )
    return result