bluecast.conformal_prediction.evaluation

Module Contents

Functions

prediction_set_coverage(→ float)	Calculate the percentyge of prediction sets that include the true label.
prediction_interval_coverage(→ Dict[float, float])	Calculate the percentage of prediction intervals that cover the true value.

bluecast.conformal_prediction.evaluation.prediction_set_coverage(y_true: numpy.ndarray | pandas.Series, prediction_sets: pandas.Series | pandas.DataFrame) → float

Calculate the percentyge of prediction sets that include the true label.

This check can be used to validate that the model covers the true labels according to the alpha set during the prediction process. :param y_true: Ground truth labels. :param prediction_sets: Predicted probabilities of shape (n_samples, 1) where each row is a set of classes.

bluecast.conformal_prediction.evaluation.prediction_interval_coverage(y_true: numpy.ndarray | pandas.Series, prediction_intervals: pandas.DataFrame, alphas: List[float]) → Dict[float, float]

Calculate the percentage of prediction intervals that cover the true value.

This check can be used to validate that the model covers the true values according to the alpha set during the prediction process. :param y_true: Ground truth labels. :param prediction_intervals: DataFrame with predicted bands according to provided confidence levels. This must

contain columns of format f”{alpha}_low” and f”{1-alpha}_high” for each format.

Parameters:

alphas – List of alphas indicating which confidence levels to check