A couple of errors when building the ensemble about flaml

[sonichi]

Problem 1 will be fixed in #45
Problem 2 is because when training the final estimator in the stacked ensemble learner, the predictions from base learners are added as new features. Then the number of monotonic constraints will be unequal to the # features in the final estimator. When the selected final estimator does not have monotonic constraints this error won't appear. To make the monotonic constraints work in the final estimator, the number of monotonic constraints need to be revised based on X_train, e.g., in fit().

[flippercy]

Thank you Chi. I see what you meant for problem 2. After checking the codes, it turns out that the ensemble is built using the estimator of the best base model as the final estimator with passthrough=True, causing the problem.

A few questions and thoughts:

1. Is it possible to let users specify the final_estimator and passthrough for the ensemble, please? In practice sometimes the only meta learner can be accepted by the business is GLM. Single boosting models are OK but a boosting model of boosting models is just too complicated for the legal team and regulators. Regarding the passthrough, there is no guarantee that one way will be better than the other so perhaps it is better to let the users decide.

2. Why does only the customized lightGBM cause this error? We've also built monotonic xgboost, RF and catboost; none of them triggered the error when used as the final_estimator. Does only lightGBM do a check on the number of features vs. length of monotone_constraint?

3. With the current setting, is there a way to solve this issue? Not sure how to specify a new monotone_constraint for the lightGBM only when it is used as the final_estimator.

Appreciate your help!

[sonichi]

1. pass_through is for original features. If you set pass_through=False, the original features will not be passed and the predictions are used as features for the final estimator. That won't solve the constraint mismatching problem.
2. Possibly.
3. Override the fit() function of the customized lgbm estimator by changing the constraints before calling self._fit(). Like:

def fit(self, X_train, y_train, budget=None, **kwargs):
     # self.params['monotonic_constraints'] = ...
     self._fit(X_train, y_train, **kwargs)

[flippercy]

I understand. Passthrough is unrelated with this issue; just a side thought.

[sonichi]

I understand. Passthrough is unrelated with this issue; just a side thought.

I created a separate issue #48 for this.

[sonichi]

@flippercy I'm closing this issue now. If your problem is not solved feel free to reopen it.