Message printed to console on read.fst

[jangorecki]

When reading fst file we are getting extra message about loading data.table package. There should be an option to suppress that message.

fst::write.fst(iris, "iris.fst")
ir=fst::read.fst("iris.fst")
Loading required namespace: data.table

After investigating I found that reading fst file actually requires data.table package to be installed while DESCRIPTION defines it as Suggested depedency. Any use of data.table should be properly escaped in such case. When we try to read fst not having data.table installed we are getting following error:

fst::write.fst(iris, "iris.fst")
> ir=fst::read.fst("iris.fst")
Loading required namespace: data.table
Failed with error:  'there is no package called 'data.table''

[MarcusKlik]

Hi @jangorecki, thanks a lot for the fix!

I think at some point data.table will end up in the Imports field again, as I plan to use data.table's fast sorting capabilities to sort chunks of data that constitute one or more groups of the data-set. Together with a merge-sort algorithm (for the chunks), that would allow for out-of-memory sorting of very big tables that are stored in a fst file.

Thanks again for the corrections!

[xiaodaigh]

Together with a merge-sort algorithm (for the chunks)

I wish I know enough C++ to help. I have started work on implementing an R-code only version.

[MarcusKlik]

Hi @xiaodaigh, an R only version using fst as a backend for writing the chunks might be almost as fast as a C++ implementation!

Most of the computational work during a merge sort is done in serializing and de-serializing chunks and writing- and reading the data to disk I think and the actual sorting of the chunks themselves (using data.table) will probably take less time.

You'll have to coordinate your workers however, and that will be relatively slow (especially on Windows :-))