Implemented multi member gz decoder #43
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added a MultiDecoderReader and MultiDecoderReaderBuf to gz. I moved the read header code to a separate function to avoid code duplication. In order to keep the reader but reset the decoder a reset_data method has been added to deflate.rs and a reset method to the CrcReader. Tests for multimember gz files are in the test directory.
|
@veldsla I'm so sorry this sat here for so long! I had no idea this was here :( In any case this looks excellent, thanks so much for implementing this! In the future feel free to ping me on any PR that sits for awhile! |
|
Haha, that's great. You did reference this PR in #44, so I just assumed you didn't want to add more functionality. Oh well assumptions are...😄 |
|
Weird... I maybe accidentally archived the email? |
|
@alexcrichton - any chance we could get a crates.io release w/ this functionality? Would like to start encouraging downstream packages to use MultiGzDecoder. Many uses cases where MultiGz files are the norm (especially bioinformatics). |
|
@pmarks certainly! I'll do a release right now |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
You mentioned in issue #41 that you were considering creating a separate type for the multistream gzip files. The nested loop method I used to parse these files in #41 (comment) becomes a bit convoluted when parsing multiple files.
Considering my rust beginner level I used it as an exercise. It works by calling finish_member instead of finish and checking if there are more bytes remaining in the buffer. If this is the case a new header read is attempted and the decoder is reset for the next member. Would this be what you had in mind?