Create Dataset from Arrow format

[jameslamb]

Summary

Apache Arrow is an open source columnar in-memory storage format that's well-suited to tabular data. It offers efficient data loading from files or other input streams, and zero-copy data sharing between languages.

Motivation

I think that this feature could allow for faster data loading, esp. from the parquet and CSV file formats. It would also allow directly training on Arrow tables, so we might be able to avoid some data copying in language wrappers (e.g. converting to a pandas data frame or R data.frame).

pyarrow offers a fast, efficient Parquet reader. I believe that reading from Parquet files directly into Arrow, then being able to efficiently create a LightGBM Dataset from that pyarrow table, would allow for faster I/O and better memory efficiency by avoiding the need to ever create a pandas data frame: https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_table.html.

Description

I'm admittedly not very experienced with C++, so maybe others can expand this description. But basically, I think it would involve adding a LGBM_DatasetCreateFromArrow similar to LGBM_DatasetCreateFromCSV:

LightGBM/src/c_api.cpp

Line 1245 in 82e2ff7

int LGBM_DatasetCreateFromCSC(const void* col_ptr,

Arrow is a fairly heavy dependency (and pyarrow in Python / {arrow} in R, by extension), so an implementation should also explore how to make these optional for users who do not need the Arrow features.

References

There is an in-progress PR to add this feature to XGBoost: dmlc/xgboost#5667

Spark added support for Arrow as a memory representation in pyspark 3 years ago: https://arrow.apache.org/blog/2017/07/26/spark-arrow/.

[StrikerRUS]

Spark added support for Arrow as a memory representation in pyspark 3 years ago:

Does it mean that right now one can use MMLSpark (https://github.com/Azure/mmlspark) for Arrow + LightGBM (similarly to parquet #1286 (comment))?

[jameslamb]

possibly! But that definitely does not satisfy this feature. Spark is a heavy dependency that many users are unlikely to have access to.

[guolinke]

I think @shiyu1994 can help with this. He has some ideas to refine the dataset class recently.

[yalwan-iqvia]

Spark added support for Arrow as a memory representation in pyspark 3 years ago:

Does it mean that right now one can use MMLSpark (https://github.com/Azure/mmlspark) for Arrow + LightGBM (similarly to parquet #1286 (comment))?

I think it would be other way round. If LightGBM implemented datasetFromArrow, it would probably be useful to speed up / improve efficiency from within MMLSpark

[StrikerRUS]

Closed in favor of being in #2302. We decided to keep all feature requests in one place.

Welcome to contribute this feature! Please re-open this issue (or post a comment if you are not a topic starter) if you are actively working on implementing this feature.

[StrikerRUS]

For the reference: Parquet data reader implementation in XGBoost with optional Arrow dependency at compile time.
dmlc/dmlc-core#653
https://github.com/dmlc/dmlc-core/blob/5eaff7643a88949c81af1e5de11945632920bf96/CMakeLists.txt#L71-L73

[jameslamb]

Linking the eventual XGBoost implementation: dmlc/xgboost#7512

[jameslamb]

Sorry, this was locked accidentally. Just unlocked it.

[jameslamb]

Linking some related PRs:

• [python-package] Allow to pass Arrow table as training data #6034
• [ci] Fix typo in dependencies #6100
• [ci] use mamba instead of conda in macOS and Linux CI jobs #6140
• [python-package] Allow to pass Arrow array as labels #6163
• [python-package] Allow to pass Arrow array as groups #6166
• [python-package] Allow to pass Arrow table and array as init scores #6167
• [python-package] Allow to pass Arrow table for prediction #6168
• [python-package] Add tests for passing Arrow arrays with empty chunks #6210

And this related discussion about how Arrow support enables polars support: #6204