bluecast.experimentation.tracking

Module Contents

Classes

ExperimentTracker

DuckDB-based implementation of ExperimentTracker used in BlueCast

class bluecast.experimentation.tracking.ExperimentTracker(db_path: str | None = None)

Bases: bluecast.config.base_classes.BaseClassExperimentTracker

DuckDB-based implementation of ExperimentTracker used in BlueCast and BlueCastCV pipelines. This triggers automatically as long as the default Xgboost model is used. For custom ml models users need to create an own Tracker. The base class from bluecast.config.base_classes can be used as an inspiration.

Uses separate tables for hyperparameter tuning and evaluation data for better organization and query performance.

property experiment_id: List[int]

Legacy compatibility property.

_init_database() → None

Initialize database schema with separate tables for different data types.

add_results(experiment_id: int, score_category: Literal[simple_train_test_score, cv_score, oof_score], training_config: bluecast.config.training_config.TrainingConfig, model_parameters: Dict[Any, Any], eval_scores: float | int | None, metric_used: str, metric_higher_is_better: bool) → None

Add an individual experiment result into the tracker.

Parameters:

• experiment_id – Sequential id. Make sure add an increment.

• score_category – Chose one of [“simple_train_test_score”, “cv_score”, “oof_score”]. “simple_train_test_score” is the default where a simple train-test split is done. “cv_score” is called when cross validation has been enabled in the TrainingConfig.

• training_config – TrainingConfig instance from bluecast.config.training_config.

• model_parameters – Dictionary with parameters of ml model (i.e. learning rate)

• eval_scores – The actual score of the experiment.

• metric_used – The name of the eval metric.

• metric_higher_is_better – True or False.

retrieve_results_as_df(table: str | None = None) → pandas.DataFrame

Convert ExperimentTracker information into a Pandas DataFrame.

Parameters:

table – Which table to query (“hyperparameter_experiments”, “evaluation_experiments”, or None for both)

Returns:

Pandas DataFrame with experiment results

get_best_score(target_metric: str, table: str | None = None) → int | float

Get the best score for a target metric.

Parameters:

• target_metric – The metric to find the best score for

• table – Which table to query (“hyperparameter_experiments”, “evaluation_experiments”, or None for both)

Returns:

Best score value

get_hyperparameter_results() → pandas.DataFrame

Get only hyperparameter tuning results.

get_evaluation_results() → pandas.DataFrame

Get only evaluation results.

get_experiment_summary() → Dict[str, Any]

Get a summary of experiments stored in the tracker.

Returns:

Dictionary with experiment statistics

close() → None

Close database connection and cleanup temporary files if created.

__del__()

Cleanup on deletion.