statmixedml / lightgbmlss

An extension of LightGBM to probabilistic modelling

LightGBMLSS - An extension of LightGBM to probabilistic modelling

We introduce a comprehensive framework that models and predicts the full conditional distribution of a univariate target as a function of covariates. Choosing from a wide range of continuous, discrete, and mixed discrete-continuous distributions, modelling and predicting the entire conditional distribution greatly enhances the flexibility of LightGBM, as it allows to create probabilistic forecasts from which prediction intervals and quantiles of interest can be derived.

Features

✅ Estimation of all distributional parameters.
✅ Normalizing Flows allow modelling of complex and multi-modal distributions.
✅ Mixture-Densities can model a diverse range of data characteristics.
✅ Zero-Adjusted and Zero-Inflated Distributions for modelling excess of zeros in the data.
✅ Automatic derivation of Gradients and Hessian of all distributional parameters using PyTorch.
✅ Automated hyper-parameter search, including pruning, is done via Optuna.
✅ The output of LightGBMLSS is explained using SHapley Additive exPlanations.
✅ LightGBMLSS provides full compatibility with all the features and functionality of LightGBM.
✅ LightGBMLSS is available in Python.

News

💥 [2024-01-19] Release of LightGBMLSS to PyPI.
💥 [2023-08-28] Release of v0.4.0 introduces Mixture-Densities. See the release notes for an overview.
💥 [2023-07-20] Release of v0.3.0 introduces Normalizing Flows. See the release notes for an overview.
💥 [2023-06-22] Release of v0.2.2. See the release notes for an overview.
💥 [2023-06-15] LightGBMLSS now supports Zero-Inflated and Zero-Adjusted Distributions.
💥 [2023-05-26] Release of v0.2.1. See the release notes for an overview.
💥 [2023-05-23] Release of v0.2.0. See the release notes for an overview.
💥 [2022-01-05] LightGBMLSS now supports estimating the full predictive distribution via Expectile Regression.
💥 [2022-01-05] LightGBMLSS now supports automatic derivation of Gradients and Hessians.
💥 [2022-01-04] LightGBMLSS is initialized with suitable starting values to improve convergence of estimation.
💥 [2022-01-04] LightGBMLSS v0.1.0 is released!

Installation

To install the development version, please use

pip install git+https://github.com/StatMixedML/LightGBMLSS.git

For the PyPI version, please use

pip install lightgbmlss

Available Distributions

Our framework is built upon PyTorch and Pyro, enabling users to harness a diverse set of distributional families. LightGBMLSS currently supports the following distributions.

How to Use

Please visit the example section for guidance on how to use the framework.

Documentation

For more information and context, please visit the documentation.

Feedback

We encourage you to provide feedback on how to enhance LightGBMLSS or request the implementation of additional distributions by opening a new discussion.

How to Cite

If you use LightGBMLSS in your research, please cite it as:

@misc{Maerz2023,
  author = {Alexander M\"arz},
  title = {{LightGBMLSS: An Extension of LightGBM to Probabilistic Modelling}},
  year = {2023},
  note = {GitHub repository, Version 0.4.0},
  howpublished = {\url{https://github.com/StatMixedML/LightGBMLSS}}
}

Reference Paper

Star History

lightgbmlss's Issues

ETA for code

Multi-task learning and ONNX support

Thanks for the great work!

I have two questions:

1. Can we perform multi-task learning in a single training where one task is classification and the target variables are categorical (classes) and the other task is regression where the target values are continuous?

2. Does LightGBMLSS models support ONNX conversions?

Prediction of quantiles for parametric distributions

For parametric distributions, I understand the model learns the mapping from data to parameters, so the model predicts the parameters for each observation.

When quantiles are predicted, the model first samples from the parametric distribution with those parameters, and then constructs quantiles from those drawn samples:
https://github.com/StatMixedML/LightGBMLSS/blob/master/lightgbmlss/distributions/distribution_utils.py#L415-L421

Is it infeasible or a reason to not instead use the theoretical quantiles for those fitted parameters directly, without sampling?

Essentially using torch_dist.icdf:

1) Fit the distribution with lightgbmlss.

2) Predict parameters for some dataset

    pred_params = distribution.predict(
        test_X,
        pred_type="parameters",
    )

3) Take the fitted parameters and instantiate a torch distribution object.

def instantiate_torch_distribution(distribution, pred_params: pd.DataFrame):
    """Prepare a torch distribution object (tensor) for the observations in the test set

    distribution
        This is an object of type lightgbmlss.model.LightGBMLSS, for which a mapping from
        features to distributional parameters was learned.
    pred_params:
        A dataframe with one column per predicted distributional param, and one row
        for each test observation that had that param predicted.
    """

    # A list of named parameters, e.g. ['loc', 'scale'] that the torch distribution
    # expects for instantiation.
    dist_arg_names = distribution.dist.distribution_arg_names

    preds_transformed = []
    for dist_arg_name in dist_arg_names:
        # predict() above already returns the values on the transformed scale
        # (e.g. the model predicts log_variance and then transforms it via exp(log_variance)
        # and this is the result from predict() above.
        pred_tensor = torch.tensor(
            pred_params[dist_arg_name].values,
            dtype=torch.float64,
        ).reshape(-1, 1)
        preds_transformed.append(pred_tensor)
    
    dist_kwargs = dict(zip(dist_arg_names, preds_transformed))
    torch_dist_fit = distribution.dist.distribution(**dist_kwargs)
    return torch_dist_fit

4) Use the torch distribution object to get the theoretical quantiles for the given parameters

def get_quantiles(q, torch_dist_fit, test_y):
    """get the qth quantiles of target values test_y

    q in [0.0, 1.0]
    """
    qt = torch.tensor([q]*test_y.size).reshape(-1, 1)
    quants=torch_dist_fit.icdf(qt)
    return quants

Misc

Separately from the above, it also looks like the samples are drawn when predict is called, even if the samples are not used in any of the returns, specifically pred_type == "parameters" and pred_type == "expectiles":
https://github.com/StatMixedML/LightGBMLSS/blob/master/lightgbmlss/distributions/distribution_utils.py#L402-L410

I can see how it makes the code cleaner if not littered with conditionals for whether to sample, and perhaps the extra computations from sampling without returning the samples is insignificant.

kwargs from the lightgbm predict are not available with the LightGBMLSS.predict

For model governance reason I want to be able to run tree by tree using the start_iteration and num_iteration arguments to see if there is a number of over fitted trees towards the end of the training.

Would it be possible to add in the **kwargs to the predict all the way through to the lightgbm.predict.

Monotonicity of expectiles

Hi,

Thanks for the great package.

I am trying to predict the expectiles of a distribution. I understood that because this approach models the entire distribution, and because the true expectiles of any distribution should be monotonic, the predictions would also be monotonic. However, I'm not getting this property. Why might this be / is there anyway to force monotonicity?

Thanks

Josh

Out of sample prediction and overall questions

Hello Mr. März!

Would this program be suited for the prediction of stock market returns with the StudenT distribution shown in the examples?
Would normal Close Price data work with LightGMBLSS as well, or does it struggle with data that was not seen in the train data?
Do you plan to implement a Cauchy distribution as well at one point?
How would the out-of sample prediction work out for some timesteps into the future after the train test phase?

I think your approach to combine statistics with gradient boosting is an amazing idea. I am a huge fan of probabilistic programming.

Best regards

Matthias

SHAP interpretations from zero-adjusted gamma model

Discussed in #34

^{Originally posted by p-schaefer February 5, 2024}
I am trying to understand the SHAP outputs from a Zero-Adjusted Gamma. I'll outline my questions below:

1. SHAP scores should sum to the models predictions. In the LSS context for a ZAG, they should sum to the concentration, rate, and gate shape parameters that are output from predict(..., pred_type="parameters"), correct? I find this is not quite the case when I look at the outputs of shap.TreeExplainer(). So I assume there is some transformation happening. Could someone elaborate on what those transformations are, and what they look like.
2. This one is maybe more general, but just in terms of interpreting the SHAP values for the shape distribution parameters, in the context of the transformations. If an observation's feature has a negative SHAP score for the gate parameter, that would imply it is, on average, decreasing the probability of a 0 outcome, and a positive SHAP score would imply it is increasing the probability of a 0 outcome. Is that interpretation correct?
3. In a normal Gamma distribution, the mean can be described by multiplying concentration and rate (I think). Can multiplying the SHAP scores for concentration and rate be used as a high level descriptor of the predictors effects on the non-zero mean part of the distribution? And, how does the transformations affect the results of that?

Thank you!

[Feature request] latest lightgbm support

The latest version of lightgbm (v3.3.4) removes fobj from the train argument of the trainingAPI and does not seem to work.
Is there any plan to support the latest version of lightgbm?

Models with init_score

The current code uses init_score to inject starting values, but what about modeling problems that need to use init_score to represent an offset, e.g. in insurance, a gamma severity or poisson frequency problem where init_score would be the log(exposure)? Could this be made to get_init_score from the dataset and incorporate that somehow during the process of setting starting values? Or would you be forced to use weights of exposure and change the response variable to y/exposure?

error when using weights in lgb.Dataset

I'm getting an error when I try to use weights in lgb datasets:

from lightgbmlss.model import *
from lightgbmlss.distributions.Expectile import *
from lightgbmlss.datasets.data_loader import load_simulated_gaussian_data

import plotnine
from plotnine import *
plotnine.options.figure_size = (20, 10)

train, test = load_simulated_gaussian_data()

X_train, y_train = train.filter(regex="x"), train["y"].values
X_test, y_test = test.filter(regex="x"), test["y"].values

weight2 = train["scale"].values

dtrain = lgb.Dataset(X_train, label=y_train,weight=weight2)



lgblss = LightGBMLSS(
    Expectile(stabilization="None",              # Options are "None", "MAD", "L2".
              expectiles = [0.05, 0.95],         # List of expectiles to be estimated, in increasing order.
              penalize_crossing = True           # Whether to include a penalty term to discourage crossing of expectiles.
              )
)

param_dict = {
    "eta":                      ["float", {"low": 1e-5,   "high": 1,     "log": True}],
    "max_depth":                ["int",   {"low": 1,      "high": 10,    "log": False}],
    "num_leaves":               ["int",   {"low": 255,    "high": 255,   "log": False}],  # set to constant for this example
    "min_data_in_leaf":         ["int",   {"low": 20,     "high": 20,    "log": False}],  # set to constant for this example
    "min_gain_to_split":        ["float", {"low": 1e-8,   "high": 40,    "log": False}],
    "min_sum_hessian_in_leaf":  ["float", {"low": 1e-8,   "high": 500,   "log": True}],
    "subsample":                ["float", {"low": 0.2,    "high": 1.0,   "log": False}],
    "feature_fraction":         ["float", {"low": 0.2,    "high": 1.0,   "log": False}],
    "boosting":                 ["categorical", ["gbdt"]],
}

np.random.seed(123)
opt_param = lgblss.hyper_opt(param_dict,
                             dtrain,
                             num_boost_round=100,        # Number of boosting iterations.
                             nfold=5,                    # Number of cv-folds.
                             early_stopping_rounds=20,   # Number of early-stopping rounds
                             max_minutes=10,             # Time budget in minutes, i.e., stop study after the given number of minutes.
                             n_trials=None,              # The number of trials. If this argument is set to None, there is no limitation on the number of trials.
                             silence=False,              # Controls the verbosity of the trail, i.e., user can silence the outputs of the trail.
                             seed=123,                   # Seed used to generate cv-folds.
                             hp_seed=None                # Seed for random number generator used in the Bayesian hyperparameter search.
                             )

I get the following error:

[I 2023-06-06 22:28:24,167] A new study created in memory with name: LightGBMLSS Hyper-Parameter Optimization
/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/optuna/progress_bar.py:56: ExperimentalWarning: Progress bar is experimental (supported from v1.2.0). The interface can change in the future.
   0%|          | 00:00/10:00
[W 2023-06-06 22:28:24,552] Trial 0 failed with parameters: {'eta': 0.0007593032665095024, 'max_depth': 3, 'num_leaves': 255, 'min_data_in_leaf': 20, 'min_gain_to_split': 17.66639853363941, 'min_sum_hessian_in_leaf': 3.0862064424185954e-05, 'subsample': 0.3026741167578493, 'feature_fraction': 0.3579892008277689, 'boosting': 'gbdt'} because of the following error: ValueError('The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()').
Traceback (most recent call last):
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/optuna/study/_optimize.py", line 200, in _run_trial
    value_or_values = func(trial)
                      ^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbmlss/model.py", line 374, in objective
    lgblss_param_tuning = self.cv(hyper_params,
                          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbmlss/model.py", line 255, in cv
    self.bstLSS_cv = lgb.cv(params,
                     ^^^^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbm/engine.py", line 640, in cv
    cvfolds.update(fobj=fobj)
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbm/engine.py", line 353, in handler_function
    ret.append(getattr(booster, name)(*args, **kwargs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbm/basic.py", line 3029, in update
    grad, hess = fobj(self.__inner_predict(0), self.train_set)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbmlss/distributions/distribution_utils.py", line 88, in objective_fn
    if data.get_weight() == None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
[W 2023-06-06 22:28:24,553] Trial 0 failed with value None.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbmlss/model.py", line 404, in hyper_opt
    study.optimize(objective, n_trials=n_trials, timeout=60 * max_minutes, show_progress_bar=True)
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/optuna/study/study.py", line 425, in optimize
    _optimize(
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/optuna/study/_optimize.py", line 66, in _optimize
    _optimize_sequential(
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/optuna/study/_optimize.py", line 163, in _optimize_sequential
    frozen_trial = _run_trial(study, func, catch)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/optuna/study/_optimize.py", line 251, in _run_trial
    raise func_err
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/optuna/study/_optimize.py", line 200, in _run_trial
    value_or_values = func(trial)
                      ^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbmlss/model.py", line 374, in objective
    lgblss_param_tuning = self.cv(hyper_params,
                          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbmlss/model.py", line 255, in cv
    self.bstLSS_cv = lgb.cv(params,
                     ^^^^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbm/engine.py", line 640, in cv
    cvfolds.update(fobj=fobj)
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbm/engine.py", line 353, in handler_function
    ret.append(getattr(booster, name)(*args, **kwargs))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbm/basic.py", line 3029, in update
    grad, hess = fobj(self.__inner_predict(0), self.train_set)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pschaefer/.local/share/r-miniconda/envs/AEC_Model/lib/python3.11/site-packages/lightgbmlss/distributions/distribution_utils.py", line 88, in objective_fn
    if data.get_weight() == None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Zero-and-One-Adjusted Beta

Very cool to see the addition of the zero-adjusted/inflated distributions. Would it be practically possible to have a zero-and-one-adjusted/inflated Beta, i.e. ZOIB, a Beta that allows the response to be [0, 1]? This goes towards the idea of using it with binary classification responses and getting, instead of the usual, natural p parameter of a Bernoulli, the distribution of p itself as a Beta distribution. This would be interesting to ask probabilistic questions about the predicted probability, I think. I could probably hack it by modifying the data with a small epsilon, but it ZABeta opens the door to doing that in a principled way. I think the idea would mean:

• A gate/probability, given by a Bernoulii, of a 0 or 1
• A gate/probability, given by a Bernoulii, of a 0 (or 1, if you like) given that it's 0 or 1
• The beta that models the outcome between 0 and 1 if it’s not zero or not one

Userdefine CV Class?

Hello,Great work,I see your code for hyper_paramters search where use lightgbm's CV method,I am trying to use the libarary for timeseries predict,So will you have plan to make CV method with User define CV Class,for traditional CV method always make worse performance in time series prediction.

Lower python version support

The required python version is not lower than 3.9 now, it's not convenient in some conditions.
Could u plz lower the python version requirement when using 'pip install git+https://github.com/StatMixedML/LightGBMLSS.git' to install.
Thank u.

Silent freezes

Running the housing data example (with the fix from #15, so that it runs at all), the process will silently die in the middle of hyperparameter optimization. By that I mean that the CPU usage drops to nothing and the process never moves on. No errors are reported and the step is still considered running by Jupyter. There doesn't appear to be any rhyme or reason as to which trial it will freeze on (sometimes right away, sometimes after 30+ rounds). It cannot be killed without force killing. The behavior occurs in a REPL with the same code, so I don't think it's Jupyter.

If I skip HP optimization and just train the model with empty (default) parameters and 100 rounds, it trains but the same silent freeze happens on any of the calls to predict in the example. That might be a clue that the issue is during prediction.

Interestingly, I have not seen this behavior with the XGBoost version of this library on the same example. I can run the whole thing to completion.

Not sure if this is related but before freezing I do get the occasional warnings like:

RuntimeWarning: invalid value encountered in subtract
RuntimeWarning: overflow encountered in cast

OS: MacOS 12.6
Python: 3.9.16

Multi-parameter optimization with custom loss function for probabilistic forecasting

Description

Dear community,

I am currently working in a probabilistic extension of LightGBM called LightGBMLSS that models all parameters of a distribution. This allows to create probabilistic forecasts from which prediction intervals and quantiles of interest can be derived.

The problem is that LightGBM doesn't permit do optimize over several parameters. Assume we have a Normal distribution y ~ N(µ, sigma). So far, my approach is a two-step procedure, where I first optimize µ with sigma fixed, and then optimize sigma with µ fixed and then iterate between these two.

Since this is inefficient, are there any ways of simultaneously optimize both µ and sigma using a custom loss function?

lightgbm v4.0.0?

Hi, I'm trying your package since you added normflows, could you please upgrade the gbm to v4.0.0?

I think they removed the fobj param from the train() function, and you'd have specify in params

Thanks!

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
lightgbmlss 0.3.0 requires lightgbm~=3.3.5, but you have lightgbm 4.0.0 which is incompatible.

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in :2 │
│ │
│ 1 # Train Model with optimized hyperparameters │
│ ❱ 2 lss_model = lgblss.train(lss_params, │
│ 3 │ │ │ │ │ │ lgbtrain, │
│ 4 │ │ │ │ │ │ num_boost_round=lss_params['num_boost_round'], │
│ 5 │ │ │ ) │
│ │
│ /home/algo/code/cqg/bts/spark/ml/.venv/lib/python3.9/site-packages/lightgbmlss/model.py:178 in │
│ train │
│ │
│ 175 │ │ if valid_sets is not None: │
│ 176 │ │ │ valid_sets = self.set_valid_margin(valid_sets, self.start_values) │
│ 177 │ │ │
│ ❱ 178 │ │ self.booster = lgb.train(params, │
│ 179 │ │ │ │ │ │ │ │ train_set, │
│ 180 │ │ │ │ │ │ │ │ num_boost_round=num_boost_round, │
│ 181 │ │ │ │ │ │ │ │ fobj=self.dist.objective_fn, │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: train() got an unexpected keyword argument 'fobj'

Binary Classification

Cool piece of software, could it be used for estimating probabilistic uncertainty for binary classification tasks?

General Discussion

Summary

Referencing to this issue StatMixedML/CatBoostLSS#5 (comment)

@jrzaurin Let`s continue the discussion on this repo.

Can't install on Ubuntu 20.04

I'm trying to install this package on a fresh virtual environment on an Ubuntu 20.04 server, but get the following error:

Building wheels for collected packages: lightgbmlss
  Building wheel for lightgbmlss (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/pschaefer/environments/my_env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-9665r5_t/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-9665r5_t/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-zv9atfsj
       cwd: /tmp/pip-req-build-9665r5_t/
  Complete output (6 lines):
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help

  error: invalid command 'bdist_wheel'
  ----------------------------------------
  ERROR: Failed building wheel for lightgbmlss
  Running setup.py clean for lightgbmlss
Failed to build lightgbmlss
ERROR: numba 0.56.4 has requirement numpy<1.24,>=1.18, but you'll have numpy 1.24.1 which is incompatible.

Full log below:

Collecting git+https://github.com/StatMixedML/LightGBMLSS.git
  Cloning https://github.com/StatMixedML/LightGBMLSS.git to /tmp/pip-req-build-9665r5_t
  Running command git clone -q https://github.com/StatMixedML/LightGBMLSS.git /tmp/pip-req-build-9665r5_t
Collecting ipywidgets
  Using cached ipywidgets-8.0.4-py3-none-any.whl (137 kB)
Collecting lightgbm~=3.3.1
  Using cached lightgbm-3.3.4-py3-none-manylinux1_x86_64.whl (2.0 MB)
Collecting matplotlib
  Using cached matplotlib-3.6.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.4 MB)
Collecting numpy~=1.16
  Using cached numpy-1.24.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Collecting optuna~=2.9.1
  Using cached optuna-2.9.1-py3-none-any.whl (302 kB)
Collecting pandas~=1.1
  Using cached pandas-1.5.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
Collecting plotnine~=0.8.0
  Using cached plotnine-0.8.0-py3-none-any.whl (4.7 MB)
Collecting properscoring~=0.1
  Using cached properscoring-0.1-py2.py3-none-any.whl (23 kB)
Collecting scikit-learn>=0.24.2
  Using cached scikit_learn-1.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.7 MB)
Collecting scipy
  Using cached scipy-1.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
Processing /home/pschaefer/.cache/pip/wheels/3d/c9/06/734ed80d6d61fad331974bf62017b4ea6b33488082b9f5e67e/shap-0.39.0-cp38-cp38-linux_x86_64.whl
Collecting torch~=1.13.0
  Using cached torch-1.13.1-cp38-cp38-manylinux1_x86_64.whl (887.4 MB)
Collecting tqdm
  Using cached tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
Collecting ipykernel>=4.5.1
  Using cached ipykernel-6.20.1-py3-none-any.whl (149 kB)
Collecting jupyterlab-widgets~=3.0
  Using cached jupyterlab_widgets-3.0.5-py3-none-any.whl (384 kB)
Collecting traitlets>=4.3.1
  Using cached traitlets-5.8.1-py3-none-any.whl (116 kB)
Collecting ipython>=6.1.0
  Using cached ipython-8.8.0-py3-none-any.whl (775 kB)
Collecting widgetsnbextension~=4.0
  Using cached widgetsnbextension-4.0.5-py3-none-any.whl (2.0 MB)
Collecting wheel
  Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
Collecting pillow>=6.2.0
  Using cached Pillow-9.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting fonttools>=4.22.0
  Using cached fonttools-4.38.0-py3-none-any.whl (965 kB)
Collecting pyparsing>=2.2.1
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting python-dateutil>=2.7
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting packaging>=20.0
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting contourpy>=1.0.1
  Using cached contourpy-1.0.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (295 kB)
Collecting alembic
  Using cached alembic-1.9.1-py3-none-any.whl (210 kB)
Collecting sqlalchemy>=1.1.0
  Using cached SQLAlchemy-1.4.46-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Collecting PyYAML
  Using cached PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
Collecting colorlog
  Using cached colorlog-6.7.0-py2.py3-none-any.whl (11 kB)
Collecting cmaes>=0.8.2
  Using cached cmaes-0.9.1-py3-none-any.whl (21 kB)
Collecting cliff
  Using cached cliff-4.1.0-py3-none-any.whl (81 kB)
Collecting pytz>=2020.1
  Using cached pytz-2022.7-py2.py3-none-any.whl (499 kB)
Collecting mizani>=0.7.3
  Using cached mizani-0.8.1-py3-none-any.whl (64 kB)
Collecting patsy>=0.5.1
  Using cached patsy-0.5.3-py2.py3-none-any.whl (233 kB)
Collecting descartes>=1.1.0
  Using cached descartes-1.1.0-py3-none-any.whl (5.8 kB)
Collecting statsmodels>=0.12.1
  Using cached statsmodels-0.13.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.9 MB)
Collecting joblib>=1.1.1
  Using cached joblib-1.2.0-py3-none-any.whl (297 kB)
Collecting threadpoolctl>=2.0.0
  Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting slicer==0.0.7
  Using cached slicer-0.0.7-py3-none-any.whl (14 kB)
Collecting numba
  Using cached numba-0.56.4-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.5 MB)
Collecting cloudpickle
  Using cached cloudpickle-2.2.0-py3-none-any.whl (25 kB)
Collecting typing-extensions
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting nvidia-cudnn-cu11==8.5.0.96; platform_system == "Linux"
  Using cached nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
Collecting nvidia-cublas-cu11==11.10.3.66; platform_system == "Linux"
  Using cached nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
Collecting nvidia-cuda-nvrtc-cu11==11.7.99; platform_system == "Linux"
  Using cached nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
Collecting nvidia-cuda-runtime-cu11==11.7.99; platform_system == "Linux"
  Using cached nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
Collecting debugpy>=1.0
  Using cached debugpy-1.6.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting psutil
  Using cached psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
Collecting comm>=0.1.1
  Using cached comm-0.1.2-py3-none-any.whl (6.5 kB)
Collecting matplotlib-inline>=0.1
  Using cached matplotlib_inline-0.1.6-py3-none-any.whl (9.4 kB)
Collecting tornado>=6.1
  Using cached tornado-6.2-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (423 kB)
Collecting jupyter-client>=6.1.12
  Using cached jupyter_client-7.4.8-py3-none-any.whl (133 kB)
Collecting nest-asyncio
  Using cached nest_asyncio-1.5.6-py3-none-any.whl (5.2 kB)
Collecting pyzmq>=17
  Using cached pyzmq-25.0.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
Collecting prompt-toolkit<3.1.0,>=3.0.11
  Using cached prompt_toolkit-3.0.36-py3-none-any.whl (386 kB)
Collecting stack-data
  Using cached stack_data-0.6.2-py3-none-any.whl (24 kB)
Collecting pexpect>4.3; sys_platform != "win32"
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting backcall
  Using cached backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Collecting pygments>=2.4.0
  Using cached Pygments-2.14.0-py3-none-any.whl (1.1 MB)
Collecting jedi>=0.16
  Using cached jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)
Collecting decorator
  Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB)
Collecting pickleshare
  Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting Mako
  Using cached Mako-1.2.4-py3-none-any.whl (78 kB)
Collecting importlib-resources; python_version < "3.9"
  Using cached importlib_resources-5.10.2-py3-none-any.whl (34 kB)
Collecting importlib-metadata; python_version < "3.9"
  Using cached importlib_metadata-6.0.0-py3-none-any.whl (21 kB)
Collecting greenlet!=0.4.17; python_version >= "3" and (platform_machine == "aarch64" or (platform_machine == "ppc64le" or (platform_machine == "x86_64" or (platform_machine == "amd64" or (platform_machine == "AMD64" or (platform_machine == "win32" or platform_machine == "WIN32"))))))
  Using cached greenlet-2.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (544 kB)
Collecting PrettyTable>=0.7.2
  Using cached prettytable-3.6.0-py3-none-any.whl (27 kB)
Collecting stevedore>=2.0.1
  Using cached stevedore-4.1.1-py3-none-any.whl (50 kB)
Collecting cmd2>=1.0.0
  Using cached cmd2-2.4.2-py3-none-any.whl (147 kB)
Collecting autopage>=0.4.0
  Using cached autopage-0.5.1-py3-none-any.whl (29 kB)
Collecting backports.zoneinfo; python_version < "3.9"
  Using cached backports.zoneinfo-0.2.1-cp38-cp38-manylinux1_x86_64.whl (74 kB)
Collecting palettable
  Using cached palettable-3.3.0-py2.py3-none-any.whl (111 kB)
Collecting llvmlite<0.40,>=0.39.0dev0
  Using cached llvmlite-0.39.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.6 MB)
Requirement already satisfied: setuptools in ./my_env/lib/python3.8/site-packages (from numba->shap~=0.39.0->lightgbmlss==0.1.0) (44.0.0)
Collecting entrypoints
  Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB)
Collecting jupyter-core>=4.9.2
  Using cached jupyter_core-5.1.3-py3-none-any.whl (93 kB)
Collecting wcwidth
  Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting executing>=1.2.0
  Using cached executing-1.2.0-py2.py3-none-any.whl (24 kB)
Collecting pure-eval
  Using cached pure_eval-0.2.2-py3-none-any.whl (11 kB)
Collecting asttokens>=2.1.0
  Using cached asttokens-2.2.1-py2.py3-none-any.whl (26 kB)
Collecting ptyprocess>=0.5
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting parso<0.9.0,>=0.8.0
  Using cached parso-0.8.3-py2.py3-none-any.whl (100 kB)
Collecting MarkupSafe>=0.9.2
  Using cached MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting zipp>=3.1.0; python_version < "3.10"
  Using cached zipp-3.11.0-py3-none-any.whl (6.6 kB)
Collecting pbr!=2.1.0,>=2.0.0
  Using cached pbr-5.11.1-py2.py3-none-any.whl (112 kB)
Processing /home/pschaefer/.cache/pip/wheels/7f/1a/65/84ff8c386bec21fca6d220ea1f5498a0367883a78dd5ba6122/pyperclip-1.8.2-py3-none-any.whl
Collecting attrs>=16.3.0
  Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Collecting platformdirs>=2.5
  Using cached platformdirs-2.6.2-py3-none-any.whl (14 kB)
Building wheels for collected packages: lightgbmlss
  Building wheel for lightgbmlss (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/pschaefer/environments/my_env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-9665r5_t/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-9665r5_t/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-zv9atfsj
       cwd: /tmp/pip-req-build-9665r5_t/
  Complete output (6 lines):
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help

  error: invalid command 'bdist_wheel'
  ----------------------------------------
  ERROR: Failed building wheel for lightgbmlss
  Running setup.py clean for lightgbmlss
Failed to build lightgbmlss
ERROR: numba 0.56.4 has requirement numpy<1.24,>=1.18, but you'll have numpy 1.24.1 which is incompatible.
Installing collected packages: debugpy, packaging, psutil, traitlets, comm, matplotlib-inline, tornado, entrypoints, platformdirs, jupyter-core, six, python-dateutil, pyzmq, nest-asyncio, jupyter-client, wcwidth, prompt-toolkit, executing, pure-eval, asttokens, stack-data, ptyprocess, pexpect, backcall, pygments, parso, jedi, decorator, pickleshare, ipython, ipykernel, jupyterlab-widgets, widgetsnbextension, ipywidgets, numpy, scipy, wheel, joblib, threadpoolctl, scikit-learn, lightgbm, pillow, kiwisolver, fonttools, pyparsing, cycler, contourpy, matplotlib, MarkupSafe, Mako, greenlet, sqlalchemy, zipp, importlib-resources, importlib-metadata, alembic, PyYAML, colorlog, cmaes, tqdm, PrettyTable, pbr, stevedore, pyperclip, attrs, cmd2, autopage, cliff, optuna, pytz, pandas, backports.zoneinfo, palettable, mizani, patsy, descartes, statsmodels, plotnine, properscoring, slicer, llvmlite, numba, cloudpickle, shap, typing-extensions, nvidia-cublas-cu11, nvidia-cudnn-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-runtime-cu11, torch, lightgbmlss
    Running setup.py install for lightgbmlss ... done
Successfully installed Mako-1.2.4 MarkupSafe-2.1.1 PrettyTable-3.6.0 PyYAML-6.0 alembic-1.9.1 asttokens-2.2.1 attrs-22.2.0 autopage-0.5.1 backcall-0.2.0 backports.zoneinfo-0.2.1 cliff-4.1.0 cloudpickle-2.2.0 cmaes-0.9.1 cmd2-2.4.2 colorlog-6.7.0 comm-0.1.2 contourpy-1.0.6 cycler-0.11.0 debugpy-1.6.5 decorator-5.1.1 descartes-1.1.0 entrypoints-0.4 executing-1.2.0 fonttools-4.38.0 greenlet-2.0.1 importlib-metadata-6.0.0 importlib-resources-5.10.2 ipykernel-6.20.1 ipython-8.8.0 ipywidgets-8.0.4 jedi-0.18.2 joblib-1.2.0 jupyter-client-7.4.8 jupyter-core-5.1.3 jupyterlab-widgets-3.0.5 kiwisolver-1.4.4 lightgbm-3.3.4 lightgbmlss-0.1.0 llvmlite-0.39.1 matplotlib-3.6.3 matplotlib-inline-0.1.6 mizani-0.8.1 nest-asyncio-1.5.6 numba-0.56.4 numpy-1.24.1 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 optuna-2.9.1 packaging-23.0 palettable-3.3.0 pandas-1.5.2 parso-0.8.3 patsy-0.5.3 pbr-5.11.1 pexpect-4.8.0 pickleshare-0.7.5 pillow-9.4.0 platformdirs-2.6.2 plotnine-0.8.0 prompt-toolkit-3.0.36 properscoring-0.1 psutil-5.9.4 ptyprocess-0.7.0 pure-eval-0.2.2 pygments-2.14.0 pyparsing-3.0.9 pyperclip-1.8.2 python-dateutil-2.8.2 pytz-2022.7 pyzmq-25.0.0 scikit-learn-1.2.0 scipy-1.10.0 shap-0.39.0 six-1.16.0 slicer-0.0.7 sqlalchemy-1.4.46 stack-data-0.6.2 statsmodels-0.13.5 stevedore-4.1.1 threadpoolctl-3.1.0 torch-1.13.1 tornado-6.2 tqdm-4.64.1 traitlets-5.8.1 typing-extensions-4.4.0 wcwidth-0.2.5 wheel-0.38.4 widgetsnbextension-4.0.5 zipp-3.11.0

Trying to import the package yields:

from lightgbmlss.model import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pschaefer/environments/my_env/lib/python3.8/site-packages/lightgbmlss/model.py", line 8, in <module>
    import shap
  File "/home/pschaefer/environments/my_env/lib/python3.8/site-packages/shap/__init__.py", line 12, in <module>
    from ._explanation import Explanation, Cohorts
  File "/home/pschaefer/environments/my_env/lib/python3.8/site-packages/shap/_explanation.py", line 12, in <module>
    from .utils._general import OpChain
  File "/home/pschaefer/environments/my_env/lib/python3.8/site-packages/shap/utils/__init__.py", line 1, in <module>
    from ._clustering import hclust_ordering, partition_tree, partition_tree_shuffle, delta_minimization_order, hclust
  File "/home/pschaefer/environments/my_env/lib/python3.8/site-packages/shap/utils/_clustering.py", line 4, in <module>
    from numba import jit
  File "/home/pschaefer/environments/my_env/lib/python3.8/site-packages/numba/__init__.py", line 42, in <module>
    from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
  File "/home/pschaefer/environments/my_env/lib/python3.8/site-packages/numba/np/ufunc/__init__.py", line 3, in <module>
    from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
  File "/home/pschaefer/environments/my_env/lib/python3.8/site-packages/numba/np/ufunc/decorators.py", line 3, in <module>
    from numba.np.ufunc import _internal
SystemError: initialization of _internal failed without raising an exception

IndexError: LightGBMLSS.train(valid_sets=[dataset_val]) calls set_valid_margin, which seems to expect both train+val

I call LightGBMLSS.train like this

early_stop = 20
dataset_params = {"feature_pre_filter": False}
dataset_train = lightgbm.Dataset(
    train_X,
    train_y["value"].ravel(),
    free_raw_data=False,
    params=dataset_params,
)
dataset_val = lightgbm.Dataset(
    val_X,
    val_y["value"].ravel(),
    free_raw_data=False,
    params=dataset_params,
)

distribution = lightgbmlss.model.LightGBMLSS(
    lightgbmlss.distributions.Gaussian.Gaussian(
        stabilization="None",
        response_fn="exp",
        loss_fn="nll",
    )
)

distribution.train(
    params=my_hyperparams,
    num_boost_round=9999999, # early_stopping should kick in before this
    train_set=dataset_train,
    valid_sets=[dataset_val],
    callbacks=[
        lightgbm.early_stopping(early_stop, verbose=False),
        lightgbm.log_evaluation(-1),
    ],
)

It results in this error

File ~/miniconda3/envs/package_name/lib/python3.9/site-packages/lightgbmlss/model.py:575, in LightGBMLSS.set_valid_margin(self, valid_sets, start_values)
    572 init_score_val1 = (np.ones(shape=(valid_sets1.get_label().shape[0], 1))) * start_values
    573 valid_sets1.set_init_score(init_score_val1.ravel(order="F"))
--> 575 valid_sets2 = valid_sets[1]
    576 init_score_val2 = (np.ones(shape=(valid_sets2.get_label().shape[0], 1))) * start_values
    577 valid_sets2.set_init_score(init_score_val2.ravel(order="F"))

IndexError: list index out of range

https://github.com/StatMixedML/LightGBMLSS/blob/master/lightgbmlss/model.py#L559-L579
It seems set_valid_margin expects both the training set and the validation set as parameters, but later when lgb.train is called, the same list is passed in as valid_sets, in addition to the training set via train_set as the first parameter:
https://github.com/StatMixedML/LightGBMLSS/blob/master/lightgbmlss/model.py#L171-L177

Have I misunderstood when I think this will cause the training set to be used as a validation set for early stopping (which may not be desirable) ?