fmohr / lcdb

LCDB (Learning Curve DataBase)

LCDB is a set of pre-computed learning curves of common learners on a variety of datasets. You can easily access it via an API we describe here.

Installation

pip install lcdb

Example Notebook

We strongly encourage to check out the example notebook in python/example-usage.ipynb. It contains all the code of this tutorial, and also shows how to get an overview of all the contained datasets.

Getting a Learning Curve

By default, you receive the learning curve for accuracy (see below for other metrics). Learning curves contain observations at powers of $\sqrt{2}$, i.e. 16, 24, 32, 45, 64, 91, 128, ... until 90% of the dataset size (also if this is not a power of $\sqrt{2}$. Since datasets have different sizes, the schedules have different lengths for different datasets, which is why there is no tensor view on the data.

By Dataset Name

curve = lcdb.get_curve("kr-vs-kp", "sklearn.linear_model.LogisticRegression")
anchors, scores_train, scores_valid, scores_test = curve

The first argument is the dataset name or the dataset id of openml.org. The second argument is the name of the learner.

By OpenML ID

curve = lcdb.get_curve(3, "sklearn.linear_model.LogisticRegression")
anchors, scores_train, scores_valid, scores_test = curve

For Other Metrics

The metric is the third positional argument. Here, we get the curve for log-loss

curve_log_loss = lcdb.get_curve("kr-vs-kp", "sklearn.linear_model.LogisticRegression", "logloss")

Currently supported metrics: accuracy, logloss

We maintain the full prediction vectors and probability distributions offline, so if you want to add some metric, please let us know and we will compute it and add it to the repository.

Plotting Curves

Of course, you can plot the curves with any tool you want. There are some built-ins though.

Quicky

lcdb.plot_train_and_test_curve(curve)

will give you

With your own axis objects

import matplotlib.pyplot as plt
fig, ax = plt.subplots(1, 2, figsize=(20, 4))
lcdb.plot_train_and_test_curve(curve, ax[0])
ax[0].set_ylabel("Accuracy")
lcdb.plot_train_and_test_curve(curve_log_loss, ax[1])
ax[1].set_ylabel("Log-Loss")
plt.show()

will give you

Training Times

You can also get the training times of the learners.

Times per Anchor

lcdb.get_train_times("kr-vs-kp", "sklearn.linear_model.LogisticRegression")

Plot Training Times

Quicky

lcdb.plot_train_times("kr-vs-kp", "sklearn.linear_model.LogisticRegression")

will give you

With your axis objects

fig, ax = plt.subplots()
lcdb.plot_train_times("kr-vs-kp", "sklearn.linear_model.LogisticRegression", ax)
lcdb.plot_train_times("kr-vs-kp", "SVC_linear", ax)

will give you

Meta-Features

All datasets come with pre-computed meta-features. You can retrieve these as follows:

Dataframe with Meta-Features for all Datasets

lcdb.get_meta_features()

Dictionary with Meta-Features for Specific Dataset

lcdb.get_meta_features("kr-vs-kp") # by name
lcdb.get_meta_features("3) # by openml.org id

Citing LCDB

If you use our database and find it helfpul, please cite the ECML paper:

@inproceedings{lcdb,
  title={LCDB 1.0: An Extensive Learning Curves Database for Classification Tasks},
  author={Mohr, Felix and Viering, Tom J and Loog, Marco and van Rijn, Jan N},
  booktitle = {Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, {ECML} {PKDD} 2022, Grenoble, France, September 19-24, 2022},
  year={2022}
}

Supported Learners

These are the 20 supported learners (all of them executed with standard parametrization):

• SVC_linear
• SVC_poly
• SVC_rbf
• SVC_sigmoid
• sklearn.tree.DecisionTreeClassifier
• sklearn.tree.ExtraTreeClassifier
• sklearn.linear_model.LogisticRegression
• sklearn.linear_model.PassiveAggressiveClassifier
• sklearn.linear_model.Perceptron
• sklearn.linear_model.RidgeClassifier
• sklearn.linear_model.SGDClassifier
• sklearn.neural_network.MLPClassifier
• sklearn.discriminant_analysis.LinearDiscriminantAnalysis
• sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis
• sklearn.naive_bayes.BernoulliNB
• sklearn.naive_bayes.MultinomialNB
• sklearn.neighbors.KNeighborsClassifier
• sklearn.ensemble.ExtraTreesClassifier
• sklearn.ensemble.RandomForestClassifier
• sklearn.ensemble.GradientBoostingClassifier

Currently Maintained Datasets

• 3 (kr-vs-kp)
• 6 (letter)
• 12 (mfeat-factors)
• 14 (mfeat-fourier)
• 16 (mfeat-karhunen)
• 18 (mfeat-morphological)
• 21 (car)
• 22 (mfeat-zernike)
• 23 (cmc)
• 24 (mushroom)
• 26 (nursery)
• 28 (optdigits)
• 30 (page-blocks)
• 31 (credit-g)

(pendigits)

• 36 (segment)
• 38 (sick)
• 44 (spambase)
• 46 (splice)
• 54 (vehicle)
• 57 (hypothyroid)
• 60 (waveform-5000)
• 179 (adult)
• 180 (covertype)
• 181 (yeast)
• 182 (satimage)
• 183 (abalone)
• 184 (kropt)
• 185 (baseball)
• 188 (eucalyptus)
• 273 (IMDB.drama)
• 293 (covertype)
• 300 (isolet)
• 351 (codrna)
• 354 (poker)
• 357 (vehicle_sensIT)
• 389 (fbis.wc)
• 390 (new3s.wc)
• 391 (re0.wc)
• 392 (oh0.wc)
• 393 (la2s.wc)
• 395 (re1.wc)
• 396 (la1s.wc)
• 398 (wap.wc)
• 399 (ohscal.wc)
• 401 (oh10.wc)
• 485 (analcatdata_vehicle)
• 554 (mnist_784)
• 679 (rmftsa_sleepdata)
• 715 (fri_c3_1000_25)
• 718 (fri_c4_1000_100)
• 720 (abalone)
• 722 (pol)
• 723 (fri_c4_1000_25)
• 727 (2dplanes)
• 728 (analcatdata_supreme)
• 734 (ailerons)
• 735 (cpu_small)
• 737 (space_ga)
• 740 (fri_c3_1000_10)
• 741 (rmftsa_sleepdata)
• 743 (fri_c1_1000_5)
• 751 (fri_c4_1000_10)
• 752 (puma32H)
• 761 (cpu_act)
• 772 (quake)
• 797 (fri_c4_1000_50)
• 799 (fri_c0_1000_5)
• 803 (delta_ailerons)
• 806 (fri_c3_1000_50)
• 807 (kin8nm)
• 813 (fri_c3_1000_5)
• 816 (puma8NH)
• 819 (delta_elevators)
• 821 (house_16H)
• 822 (cal_housing)
• 823 (houses)
• 833 (bank32nh)
• 837 (fri_c1_1000_50)
• 843 (house_8L)
• 845 (fri_c0_1000_10)
• 846 (elevators)
• 847 (wind)
• 849 (fri_c0_1000_25)
• 866 (fri_c2_1000_50)
• 871 (pollen)
• 881 (mv)
• 897 (colleges_aaup)
• 901 (fried)
• 903 (fri_c2_1000_25)
• 904 (fri_c0_1000_50)
• 910 (fri_c1_1000_10)
• 912 (fri_c2_1000_5)
• 913 (fri_c2_1000_10)
• 914 (balloon)
• 917 (fri_c1_1000_25)
• 923 (visualizing_soil)
• 930 (colleges_usnews)
• 934 (socmob)
• 953 (splice)
• 958 (segment)
• 959 (nursery)
• 962 (mfeat-morphological)
• 966 (analcatdata_halloffame)
• 971 (mfeat-fourier)
• 976 (JapaneseVowels)
• 977 (letter)
• 978 (mfeat-factors)
• 979 (waveform-5000)
• 980 (optdigits)
• 991 (car)
• 993 (kdd_ipums_la_97-small)
• 995 (mfeat-zernike)
• 1000 (hypothyroid)
• 1002 (ipums_la_98-small)
• 1018 (ipums_la_99-small)
• 1019 (pendigits)
• 1020 (mfeat-karhunen)
• 1021 (page-blocks)
• 1036 (sylva_agnostic)
• 1037 (ada_prior)
• 1039 (hiva_agnostic)
• 1040 (sylva_prior)
• 1041 (gina_prior2)
• 1042 (gina_prior)
• 1049 (pc4)
• 1050 (pc3)
• 1053 (jm1)
• 1059 (ar1)
• 1067 (kc1)
• 1068 (pc1)
• 1069 (pc2)
• 1111 (KDDCup09_appetency)
• 1116 (musk)
• 1119 (adult-census)
• 1120 (MagicTelescope)
• 1128 (OVA_Breast)
• 1130 (OVA_Lung)
• 1134 (OVA_Kidney)
• 1138 (OVA_Uterus)
• 1139 (OVA_Omentum)
• 1142 (OVA_Endometrium)
• 1146 (OVA_Prostate)
• 1161 (OVA_Colon)
• 1166 (OVA_Ovary)
• 1216 (Click_prediction_small)
• 1242 (vehicleNorm)
• 1457 (amazon-commerce-reviews)
• 1461 (bank-marketing)
• 1464 (blood-transfusion-service-center)
• 1468 (cnae-9)
• 1475 (first-order-theorem-proving)
• 1485 (madelon)
• 1486 (nomao)
• 1487 (ozone-level-8hr)
• 1489 (phoneme)
• 1494 (qsar-biodeg)
• 1501 (semeion)
• 1515 (micro-mass)
• 1569 (poker-hand)
• 1590 (adult)
• 4134 (Bioresponse)
• 4135 (Amazon_employee_access)
• 4136 (Dexter)
• 4137 (Dorothea)
• 4534 (PhishingWebsites)
• 4538 (GesturePhaseSegmentationProcessed)
• 4541 (Diabetes130US)
• 4552 (BachChoralHarmony)
• 23380 (cjs)
• 23512 (higgs)
• 23517 (numerai28.6)
• 40497 (thyroid-ann)
• 40498 (wine-quality-white)
• 40668 (connect-4)
• 40670 (dna)
• 40685 (shuttle)
• 40691 (wine-quality-red)
• 40701 (churn)
• 40900 (Satellite)
• 40926 (CIFAR_10_small)
• 40971 (collins)
• 40975 (car)
• 40978 (Internet-Advertisements)
• 40981 (Australian)
• 40982 (steel-plates-fault)
• 40983 (wilt)
• 40984 (segment)
• 40996 (Fashion-MNIST)
• 41026 (gisette)
• 41027 (jungle_chess_2pcs_raw_endgame_complete)
• 41064 (convex)
• 41065 (mnist_rotation)
• 41066 (secom)
• 41138 (APSFailure)
• 41142 (christine)
• 41143 (jasmine)
• 41144 (madeline)
• 41145 (philippine)
• 41146 (sylvine)
• 41147 (albert)
• 41150 (MiniBooNE)
• 41156 (ada)
• 41157 (arcene)
• 41158 (gina)
• 41159 (guillermo)
• 41161 (riccardo)
• 41162 (kick)
• 41163 (dilbert)
• 41164 (fabert)
• 41165 (robert)
• 41166 (volkert)
• 41167 (dionis)
• 41168 (jannis)
• 41169 (helena)
• 41946 (Sick_numeric)
• 42732 (sf-police-incidents)
• 42733 (Click_prediction_small)
• 42734 (okcupid-stem)

lcdb's Issues

Do not serialize configspace

Validation and test subsampling rate should be read from configuration file

[LCDB 2.0] KNNWorkflow ValueError when classes are not integers

The following command triggers a ValueError:

lcdb test -id 6 -w lcdb.workflow.sklearn.KNNWorkflow -m -vs 42 -ts 42 -ws 42 --parameters '{"metric": "minkowski", "n_neighbors": 3, "pp@cat_encoder": "ordinal", "pp@decomposition": "ka_nystroem", "pp@featuregen": "none", "pp@featureselector": "none", "pp@scaler": "minmax", "weights": "uniform", "p": 5, "pp@kernel_pca_kernel": "linear", "pp@kernel_pca_n_components": 0.25, "pp@poly_degree": 2, "pp@selectp_percentile": 25, "pp@std_with_std": true}'

Output:

Traceback (most recent call last):
  File "/Users/romainegele/Documents/Research/LCDB/lcdb/publications/2023-neurips/lcdb/controller.py", line 158, in build_curves
    self.compute_metrics_for_workflow()
  File "/Users/romainegele/Documents/Research/LCDB/lcdb/publications/2023-neurips/lcdb/controller.py", line 265, in compute_metrics_for_workflow
    predictions, labels = self.get_predictions()
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/romainegele/Documents/Research/LCDB/lcdb/publications/2023-neurips/lcdb/controller.py", line 283, in get_predictions
    keys[f"y_pred_{label_split}"] = self.workflow.predict(X_split)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/romainegele/Documents/Research/LCDB/lcdb/publications/2023-neurips/lcdb/workflow/_base_workflow.py", line 43, in predict
    y_pred = self._predict(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/romainegele/Documents/Research/LCDB/lcdb/publications/2023-neurips/lcdb/workflow/sklearn/_knn.py", line 87, in _predict
    return self.learner.predict(X)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/romainegele/Documents/Argonne/deephyper-scikit-learn/sklearn/neighbors/_classification.py", line 258, in predict
    probabilities = self.predict_proba(X)
                    ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/romainegele/Documents/Argonne/deephyper-scikit-learn/sklearn/neighbors/_classification.py", line 336, in predict_proba
    probabilities = ArgKminClassMode.compute(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/romainegele/Documents/Argonne/deephyper-scikit-learn/sklearn/metrics/_pairwise_distances_reduction/_dispatcher.py", line 579, in compute
    unique_Y_labels=np.array(unique_Y_labels, dtype=np.intp),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: 'A'

Make results a dictionary

Latinhypercube sampling

AUC computation

It would be nice if we can include AUC scores for the classifiers. This is possible also for the SVM with decision_function

The AUC computation is a bit non-trivial for the one versus rest classifiers, and need to be dealt with by hand.

For SVM, it is possible to convert one versus rest decision functions to one versus one decision functions. This can be used to get the one versus rest AUC. See here:
https://github.com/scikit-learn/scikit-learn/blob/2a2772a87b6c772dc3b8292bcffb990ce27515a8/sklearn/utils/multiclass.py#L479

Additionally, we need to deal with classes that were not seen during training.

Neurocomputing special issue on early fidelities

Technical TODOs

• pick classification datasets on which to run learning curves
• collect the number of trainable parameters as a metadata for densenn workflow
• create run functions to benchmark multi-fidelity strategies with lcdb 1.0
• show from characteristics (scores + clusters) which policy should whin (constant fidelity, SHA, LCE) and therefore which datasets/curves should be used to benchmark X-class of multi-fidelity algorithm
• present all learning curves with minimization normalization (BCE: balanced error rate for classification, MSE: mean squared error for regression)
• set number of trees in random forest as constant for learning curves of the same length @fmohr
• set the number of epochs in densenn as constant for learning curves of the same length @Deathn0t
• remove white color from colormap (can't see the curves otherwise)
• develop the "random"-predictor for classification @fmohr
• develop the "constant"-predictor for classification (mode of targets, ie always predict the most frequent class in training data) @Deathn0t
• develop the "constant"-predictor for regression (mean of targets for squared L2 loss, median for absolute L1 loss) @fmohr
• add constant and random predictor performance for regression/classification learning curves and use it to define the middle of the ranking (colormap)
• also include regression in lcdb 2.0 @fmohr
• add OOB curve for random forests @fmohr
• normalize curves with respect to the constant (classification) or constant (regression) predictor (aka, the "simplest" thing to do), this predictor should also be fitted on the "train"
• cluster "learning curves sets" and identify the properties of each cluster (eg, crossings, all good, all bad, god vs bad)

Paper TODOs

• review abstract + introduction @fmohr
• review related work section @fmohr + @tomviering (for respective work on 2 surveys)
• review method section for successive halving, learning curve extrapolation and constant fidelity @tomviering
• write the formula of the scores based on regret @tomviering
• for each benchmark describe how the validation and test splits/performances are created

Preprocessing hyperparameters

Sample default config first and add to experiment table

A few datasets from openML CC-18 are unavailable

Should look into whats wrong and try to add them to the lcdb

This is the current list of datasets used:
3, 6, 11, 12, 14, 15, 16, 18, 22, 23, 28, 29, 31, 32, 37, 38, 44, 50, 54, 151, 182, 188, 300, 307, 458, 469, 554, 1049, 1050, 1053, 1063, 1067, 1068, 1461, 1462, 1464, 1468, 1475, 1478, 1480, 1485, 1486, 1487, 1489, 1494, 1497, 1501, 1510, 1590, 4134, 4534, 4538, 23381, 23517, 40499, 40668, 40670, 40701, 40923, 40927, 40975, 40978, 40979, 40982, 40983, 40984, 40996, 41027

[LCDB 2.0] ValueError on DenseNNWorkflow

The command to reproduce is:

lcdb test -id 6 -w lcdb.workflow.keras.DenseNNWorkflow -m -vs 42 -ts 42 -ws 42 --parameters '{"activation": "tanh", "activity_regularizer": "none", "batch_norm": false, "batch_size": 74, "bias_regularizer": "L1", "dropout_rate": 0.5235265140514189, "kernel_initializer": "random_uniform", "kernel_regularizer": "none", "learning_rate": 0.0010642794778408, "num_layers": 13, "num_units": 4, "optimizer": "Nadam", "regularizer_factor": 0.7423546250656852, "shuffle_each_epoch": false, "skip_co": true, "transform_cat": "onehot", "transform_real": "none"}'

The example trace is:

2023-12-22 22:56:01,852 - ERROR - controller.py:fit_workflow_on_current_anchor - Error while fitting the workflow:
Traceback (most recent call last):
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/lcdb/controller.py", line 208, in fit_workflow_on_current_anchor
    self.workflow.fit(
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/lcdb/utils.py", line 67, in terminate_on_timeout
    return results.get(timeout)
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/build/dhenv/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/build/dhenv/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/lcdb/workflow/_base_workflow.py", line 31, in fit
    self._fit(X=X, y=y, metadata=metadata, *args, **kwargs)
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/lcdb/workflow/keras/_dense.py", line 299, in _fit
    y_valid_ = self._transformer_label.transform(y_valid)
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/build/dhenv/lib/python3.10/site-packages/sklearn/preprocessing/_label.py", line 137, in transform
    return _encode(y, uniques=self.classes_)
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/build/dhenv/lib/python3.10/site-packages/sklearn/utils/_encode.py", line 232, in _encode
    raise ValueError(f"y contains previously unseen labels: {str(diff)}")
ValueError: y contains previously unseen labels: [1, 3, 4, 6, 7, 9, 11, 13, 14, 16, 18, 19, 24]

Integrate preprocessing into workflow

Running example-usage.ipynb does not work

Running
https://github.com/fmohr/lcdb/blob/main/python/example-usage.ipynb
does not work and results in errors when lcdb is freshly installed via pip install.

Sampling of hyperparameters should be done differently for each dataset

Check validity of LHS

[LCDB 2.0] RandomForestWorkflow ValueError for OOB Scores

The following command triggers a ValueError:

lcdb test -id 6 -w lcdb.workflow.sklearn.RandomForestWorkflow -m -vs 42 -ts 42 -ws 42 --parameters '{"bootstrap": false, "criterion": "log_loss", "max_features": "all", "max_samples": 0.6460461826006697, "min_impurity_decrease": 0.7762379021238405, "min_samples_leaf": 12, "min_samples_split": 24, "n_estimators": 2000, "pp@cat_encoder": "onehot", "pp@decomposition": "none", "pp@featuregen": "poly", "pp@featureselector": "selectp", "pp@scaler": "std", "pp@kernel_pca_kernel": "linear", "pp@kernel_pca_n_components": 0.25, "pp@poly_degree": 2, "pp@selectp_percentile": 98, "pp@std_with_std": true}'

Output:

Traceback (most recent call last):
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/lcdb/controller.py", line 208, in fit_workflow_on_current_anchor
    self.workflow.fit(
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/lcdb/utils.py", line 67, in terminate_on_timeout
    return results.get(timeout)
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/build/dhenv/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/build/dhenv/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/lcdb/workflow/_base_workflow.py", line 31, in fit
    self._fit(X=X, y=y, metadata=metadata, *args, **kwargs)
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/lcdb/workflow/sklearn/_randomforest.py", line 186, in _fit
    scorer.score(
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/lcdb/scorer.py", line 80, in score
    roc_auc_score(
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/build/dhenv/lib/python3.10/site-packages/sklearn/utils/_param_validation.py", line 214, in wrapper
    return func(*args, **kwargs)
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/build/dhenv/lib/python3.10/site-packages/sklearn/metrics/_ranking.py", line 621, in roc_auc_score
    return _multiclass_roc_auc_score(
  File "/lus/grand/projects/datascience/regele/polaris/lcdb/publications/2023-neurips/build/dhenv/lib/python3.10/site-packages/sklearn/metrics/_ranking.py", line 694, in _multiclass_roc_auc_score
    raise ValueError(
ValueError: Target scores need to be probabilities for multiclass roc_auc, i.e. they should sum up to 1.0 over classes

Installing the package and then importing does not work

I first successfully installed the package using pip install lcdb. However, when I try to import the library, I get the error: ModuleNotFoundError: No module named 'openml'. I believe this error may be due to the fact that the setup.py does not have openml in the install_requires array,