[R-package] Allow data to live in memory mapped file

[nbenn]

In R, two packages exists that allow for working with matrices as memory mapped files: bigmemory and more recently bigstatsr. As the underlying data structures are regular column major arrays, it should be not hard to extend the LightGBM R-package to process such arrays in similar fashion as the already implemented base::matrix() objects.

I'll put together a proof of concept and I'm opening this issue to discuss whether this could be of interest for the wider LightGBM/R community.

[nbenn]

A proof of concept is available from master...nbenn:issue-#2184

To run, a patched version of bigstatsr is needed, available as

devtools::install_github("nbenn/bigstatsr@data-ptr")

[jameslamb]

Thanks @nbenn ! I'll take a look in the next few days. Appreciate you making a PoC

[StrikerRUS]

Closed in favor of being in #2302. We decided to keep all feature requests in one place.

Welcome to contribute this feature! Please re-open this issue (or post a comment if you are not a topic starter) if you are actively working on implementing this feature.