[dask] Add support for early stopping in Dask interface

[jameslamb]

Summary

DaskLGBMClassifier and DaskLGBMRegressor in the Python package should support early stopping.

Motivation

Early stopping is generally useful with gradient boosting algorithms, to avoid wasted training iterations or unnecessary growth in the model size once desirable performance has been achieved. This feature is available in the non-Dask interfaces for LightGBM, and should be available with the Dask one.

Description

This should mimic the approach XGBoost took (https://github.com/dmlc/xgboost/blob/516a93d25c3b6899558700430ffc99a29ea21e1a/python-package/xgboost/dask.py#L1386), where eval_set contains Dask collection s(Dask Array or Dask DataFrame).

References

• relevant XGBoost PR (see linked issues and PRs): [FEA] DaskXGBClassifier Early Stopping dmlc/xgboost#5495
• one proposal: [python] [dask] add initial dask integration #3515 (comment), h2oai/dask-lightgbm@8cc8e83

[jameslamb]

Closing in favor of being in #2302 with other feature requests. Please leave a comment here if you'd like to work on this.

[ffineis]

Hey I'd like to take this if it's cool - planning mainly base this off of changes called out in #3515 and try to bring this in line with xgboost.dask.

[jameslamb]

sure, thank you! I'm really close to having a small reproducible example for the random "cannot bind to port XXXX" issue, will link that here when I've written it up.

[ffineis]

sure, thank you! I'm really close to having a small reproducible example for the random "cannot bind to port XXXX" issue, will link that here when I've written it up.

Omigod lifesaver, thank you!

[jameslamb]

@ffineis do you think you'll have time to work on this this week? We're planning to do a 3.2.0 release of LightGBM in the next week or two. I didn't include this in my list of must-have Dask features for the next release (#3872 (comment)), but I'd love to try to get this change in if we can since it can have such a big impact on training runtime.

If you don't have time this week, could I take this back from you and try it out?

Thanks so much for all your help with the Dask module so far!!

[ffineis]

Hey! Yes sorry I’ve started planning for this, gimme another week? The only tricky part of matching xgboost.dask’s implementation is the id(training data) check. So yeah, sorry haven’t made any progress yet but was planning to make some commits this week.

…

On Sat, Feb 6, 2021 at 5:19 PM James Lamb ***@***.***> wrote: @ffineis <https://github.com/ffineis> do you think you'll have time to work on this this week? We're planning to do a 3.2.0 release of LightGBM in the next week or two. I didn't include this in my list of must-have Dask features for the next release (#3872 (comment) <#3872 (comment)>), but I'd love to try to get this change in if we can since it can have such a big impact on training runtime. If you don't have time this week, could I take this back from you and try it out? Thanks so much for all your help with the Dask module so far!! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3712 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACZIXCNZCJFB4DG5IQV7SCTS5XFBPANCNFSM4VSHUQIQ> .

[jameslamb]

oh yeah no problem, thanks!

[jmoralez]

Hi. I've been playing around with this and have a working (although horrible) implementation of this. The approach I took was using the futures of the persisted collections instead of turning the collections to lists of delayeds, so this avoids recomputing the training set in case it is in the eval_set as well (because both futures have the same key). Would you guys be interested in discussing this approach?

[jameslamb]

Interesting! I'll leave it to @ffineis to comment on that. Right now, I think our highest priority is supporting early stopping, and it would be ok if the first implementation of that merged to master computes the training data twice.

[ffineis]

Hey @jmoralez, thanks for the ideation! Honestly, don't feel bad, I think any implementation of ES will be pretty hairy given that we're using lists of delayed partitions instead of distributed lgbm.Datasets. I'm a fan of how xgboost.dask attempts to accomplish what you've mention via id - I'm just going to check whether each X in a (X, y) eval set shares the same id as that of data. If so, this will instruct _train_part (local worker training function) to just use the local data in for any eval set's parts that had originally called for data, so hoping we'll avoid re-computing the entire trainset.

This method works if the entirety of an eval X equals data, but I'm guessing your approach goes even further in that it would also include cases when an eval X is a subset of partitions of training data? I'd say in the spirit of how PR's have been going (modular, one change at a time) - hold off until a naive early stopping approach gets merged, but then open an issue and follow up with an improvement PR - does that work?

[jmoralez]

Yeah that sounds fair. Is there a plan to create an lgb.DaskDataset?

[ffineis]

@jmoralez my guess is yes - I think check out #2302? Jlamb has a running list of dask features somewhere.

PR for this coming tomorrow night.

[jameslamb]

yes but I haven't written it up. Will do that right now. I like what xgboost did with DaskDMatrix. https://github.com/dmlc/xgboost/blob/9a0399e8981b2279d921fe2312f7ab1b880fd3c3/python-package/xgboost/dask.py#L185

[jameslamb]

I've added #3944 to track the idea of a DaskDataset. That's a fairly invasive change so I don't think it should be done before the 3.2.0 release (#3872)

[StrikerRUS]

Closing for now due to the lack of active work on this feature and open PRs.