Binary Classification about lightgbmlss

[StatMixedML]

@firmai Thanks for your interest in the project.

As of now, it is implemented for regression tasks only, since classification tasks are probabilistic already, assigning probabilities to each class.

[ihopethiswillfi]

Agreed. What I'd personally be interested in is a classifier which can output calibrated probabilities in one go. Then you don't have to fit a calibrator (e.g. sklearn CalibratedClassifier) on top of the already trained model. I am not sure if this is even possible by the way. Just throwing it out here because @StatMixedML you really seem to know what you are doing :)

[StatMixedML]

@ihopethiswillfi Thanks for the clarification.

I definitely do see the value of probability calibration and for it being a valuable tool for evaluating any classifier, but I am no sure I how you would get that out from the specified likelihood.

Both XGBoostLSS and LightGBMLSS are trained to learn the parameters of a specified likelihood. So for the binary-case it would be a Bernoulli, and for the multi-class classification a multi-nomial distribution. For the Bernoulli, there is only one parameter p (q=1-p) that the model would learn. I don't see how to extend that to allow for a calibration, but very happy to hear your comments on this. Would be a nice extension.

[ihopethiswillfi]

Unfortunately I have no idea either. Thanks for the quick response.

(by the way I'm not OP)

[StatMixedML]

@firmai Can we close the issue?

[neverfox]

I was thinking about (probabilistic) classification problems in the context of a meta-model for the distribution of Bernoulii p itself (perhaps a Beta). Yes, when you get a probability from a classifier, it's Bernoulii and you can sample from it. But what if the probability is 1 (or 0)? Your samples will all be 1 (or 0) and they would have 0 variance. What I think would be interesting would be the ability to sample from the distribution of p rather than the (Bernoulli) distribution parameterized by p. That would, I think, let you examine questions about what your model might have predicted, within some credibility bounds. You'd be able to talk about the confidence of the model's predictions rather than the confidence in the model's outcomes given that the predictions are correct. Am I thinking about that correctly?