Add compression algorithms facility to file IO #84
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
nicholasjng
force-pushed
the
add-basic-compressions
branch
from
60fefaa
to
57429fd
Compare
maxmynter
approved these changes
Feb 22, 2024
Works similarly to the file driver lookup, except the compression can also be None, in which case a normal file descriptor is used. Canonicalizes the extension of the resulting file regardless of which file name is used based on driver and compression algorithm. This might be surprising behavior, but simplifies driver and compression inference significantly on roundtrips. Scraps the custom registries in FileIO constructor, because we expose reg/dereg hooks for file drivers and compressions.
nicholasjng
force-pushed
the
add-basic-compressions
branch
from
57429fd
to
6dce1a5
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Still work in progress, I'm currently debating which other compressions to add (lzma? xz? ...), and whether to expose the stateful hooks (get_driver_implementation() | get_compression_algorithm()) right now instead of letting people roll their own SerDe registry.Works similarly to the file driver lookup, except the compression can
also be None, in which case a normal file descriptor is used.
Canonicalizes the extension of the resulting file regardless of which
file name is used based on driver and compression algorithm. This might
be surprising behavior, but simplifies driver and compression inference
significantly on roundtrips.
Scraps the custom registries in FileIO constructor, because we intend to
expose reg/dereg hooks for the default later on.
Available compressions are gzip and bz2, which are cross-platform Python stdlib builtins.