ja-thomas / pbmohpo

Preferential Bayesian Multi-Objective Hyperparameter Optimization

Preferential Bayesian Multi-Objective Hyperparameter Optimization

Documentation

https://ja-thomas.github.io/pbmohpo/

Summary

While hyperparameter optimization has been accepted as an important component of a machine learning task, it is often conducted in an unrealistic setting.

While research often presents Machine Learning as a one dimensional problem with a single evaluation criterion like Accuracy, real-world applications seldom present in that way: Multiple - often conflicting - performance metrics are of interest to a decision maker (DM), thus making the decision for a fully configured model often more challenging as a suitable trade-off needs to be identified.

While this problem can be solved via expensive multi-objective black-box optimization, a DM might in reality not be able to specify all of their evaluation criteria, but simply state their preference of one model over another or produce a ranking of models given a list.

Other applications like A/B testing or recommender systems similarly only provide feedback this way.

This scenario of optimizing only through pairwise preferences has been explored in K-armed duelling bandit problems and Preferential Bayesian Optimization (PBO). While PBO gives a direct answer to expensive black-box optimization based on pairwise preferences, the methods introduce a second bottleneck in the iterative process of Bayesian Optimization (BO).

In addition to the expensive evaluation(s) of selected models, the optimization now needs to wait for a preference expression from the DM before selecting new models to evaluate. To avoid unnecessary idle time and create a drawn-out optimization process, mechanisms need to be developed in order to use computation resources when available as well as use DM time to rank models when the DM is available.

pbmohpo's Issues

gh-pages does not deploy correclty

archive visualization

YAHPO problem wrapper changes dict representation of a config after evaluating that config

Somewhat weird bug:

from pbmohpo.problems.yahpo import YAHPO

problem = YAHPO("iaml_xgboost", instance = "40981", objective_names = ["mmce", "nf"])
problem.get_config_space()

problem.fix_hps = {"booster": "gbtree", "trainsize" : 1}
cs = problem.get_config_space()
cs.seed(0)
x = cs.sample_configuration(1)

# looks as expected
x.get_dictionary()

{'alpha': 0.6945356257424766,
 'colsample_bylevel': 0.7180374727086953,
 'colsample_bytree': 0.6067357423109274,
 'eta': 0.01511933646764101,
 'gamma': 0.011288832727483048,
 'lambda': 3.3209228886981608,
 'max_depth': 7,
 'min_child_weight': 97.18110224911669,
 'nrounds': 1569,
 'subsample': 0.4450973669431999}

problem(x)

{'mmce': -0.49803417921066284, 'nf': -0.0}

x.get_dictionary()
# now also includes the fixed hyperparameters from `problem.fix_hps`  which is unintended behavior

{'alpha': 0.6945356257424766,
 'colsample_bylevel': 0.7180374727086953,
 'colsample_bytree': 0.6067357423109274,
 'eta': 0.01511933646764101,
 'gamma': 0.011288832727483048,
 'lambda': 3.3209228886981608,
 'max_depth': 7,
 'min_child_weight': 97.18110224911669,
 'nrounds': 1569,
 'subsample': 0.4450973669431999,
 'task_id': '40981',
 'booster': 'gbtree',
 'trainsize': 1}

to fix this we should work with a deepcopy within the YAHPO wrapper https://github.com/ja-thomas/pbmohpo/blob/d0e5ae3db39cad3b9f8068bae1fbe57ddc2ca6ea/pbmohpo/problems/yahpo.py#LL68C1-L68C1

add tests of remove_hp_from_cs to make sure that this does work consistently