I/O metaissuse

[Keno]

1. Bikeshed socket API
2. Write buffering
3. Performance improvements for reading (probably needs I/O benchmarks)
4. Complete UV based file API (preferably after Struct patch merge)
5. Move initial I/O detection into julia and make STDIO object appropriate julia objects accordingly (depends on 4)

[ViralBShah]

This list sounds so juicy - looking forward to seeing how it all works when it is done.

[kmsquire]

I've been looking forward to this as well! One idea I discussed briefly with @StefanKarpinski was to have transparent reading/writing of compressed (gzip, bzip2, xz) files (with a way to access the raw stream if needed). The current gzip interface is somewhat limited, so it would be nice to integrate with/build on the I/O system once this work has moved forward.

For gzip and xz, http://git.tukaani.org/?p=libxzfile.git might be worth looking at.

[dmbates]

+1

A binary, saved R dataset (the RDX2 format) can be compressed with gzip, bzip2 or xz and xz is preferred for very large datasets. Right now I need to write them out with gzip compression and use

using DataFrames, RDA
require("gzip")
using GZip

ff = gzopen(<filename>)
fr = read_RDA(ff)
close(ff)

[johnmyleswhite]

Yes, this would be great to have.

-- John

On Jan 25, 2013, at 12:45 PM, dmbates [email protected] wrote:

+1

A binary, saved R dataset (the RDX2 format) can be compressed with gzip, bzip2 or xz and xz is preferred for very large datasets. Right now I need to write them out with gzip compression and use

using DataFrames, RDA
require("gzip")
using GZip

ff = gzopen()
fr = read_RDA(ff)
close(ff)
—
Reply to this email directly or view it on GitHub.

[JeffBezanson]

Highest priority is eliminating the need for the Proceed with caution! warning.

[ViralBShah]

Where do we stand on this one w.r.t. 0.2?

[JeffBezanson]

Subsumed by #3887.