bluecast.experimentation.tracking

Module Contents

Classes

ExperimentTracker

Default implementation of ExperimentTracker used in BlueCast

class bluecast.experimentation.tracking.ExperimentTracker

Bases: bluecast.config.base_classes.BaseClassExperimentTracker

Default implementation of ExperimentTracker used in BlueCast and BlueCastCV pipelines. This triggers automatically as long as the default Xgboost model is used. For custom ml models ueers need to create an own Tracker. The base class from bluecast.config.base_classes can be used as an inspiration.

add_results(experiment_id: int, score_category: Literal[simple_train_test_score, cv_score, oof_score], training_config: bluecast.config.training_config.TrainingConfig, model_parameters: Dict[Any, Any], eval_scores: float | int | None, metric_used: str, metric_higher_is_better: bool) → None

Add an individual experiment result into the tracker.

Parameters:

• experiment_id – Sequential id. Make sure add an increment.

• score_category – Chose one of [“simple_train_test_score”, “cv_score”, “oof_score”]. “simple_train_test_score” is the default where a simple train-test split is done. “cv_score” is called when cross validation has been enabled in the TrainingConfig.

• training_config – TrainingConfig instance from bluecast.config.training_config.

• model_parameters – Dictionary with parameters of ml model (i.e. learning rate)

• eval_scores – The actual score of the experiment.

• metric_used – The name of the eval metric.

• metric_higher_is_better – True or False.

retrieve_results_as_df() → pandas.DataFrame

Convert ExperimentTracker information into a Pandas DataFrame.

In the default implementation this contains TrainingConfig, XgboostConfig, hyperparameters, eval metric and score.

get_best_score(target_metric: str) → int | float

Expects results in the tracker