Comparison of GB & loss policies about hep_ml

[arogozhnikov]

Check on some weighted dataset several options:

• sklearn (logloss, exploss)
• xgboost (logloss)
• UGBoost
  • logloss, exploss only with target = gradient, weight = 1
  • logloss, exploss with target = gradient/hessian, weight = hessian

[arogozhnikov]

todo: find out why there is such difference in variations of predictions

[arogozhnikov]

Some experiments I've done long ago say that:

• UGradientBoosting update policy (all data + regularization) fails on the start, but turns out to be more stable when hundreds or thousands of trees are used.
• see also discussion on kaggle, which confirms this point.

As of last point (setting target and weight), there seems to be no strong winner.
Mostly following tradition, I'll prefer grad/hess version.