Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support HDF5/JLD/JLD2 format #23

Open
oxinabox opened this issue Oct 3, 2019 · 1 comment
Open

Support HDF5/JLD/JLD2 format #23

oxinabox opened this issue Oct 3, 2019 · 1 comment
Labels
enhancement New feature or request

Comments

@oxinabox
Copy link
Member

oxinabox commented Oct 3, 2019

While none of these formats support directly streaming, and need to file
we could always work around that by creating tempfiles,
then reading them back in.

And that will be fine since most linux puts TMPDIR on a ramdrive right?

this is how I used to save data in the JLD, on SwiftObjectStorage
https://github.com/oxinabox/SwiftObjectStores.jl/blob/45acf6e5b9446672674964122763528a4d452322/src/upload.jl#L42-L51

@rofinn
Copy link
Member

rofinn commented Oct 3, 2019

We could support that, but it won't be useful for our use case on AWS Batch. On batch, we don't want large temp files in our containers as this will complicate our configuration and result in blocking on IO more often.

@nickrobinson251 nickrobinson251 added the enhancement New feature or request label Mar 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants