Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ZstandardFile class (and "open" method) that supports 'r' and 'w' modes, just like in Python's stdlib #118

Closed
d33tah opened this issue Jul 31, 2020 · 2 comments

Comments

@d33tah
Copy link

d33tah commented Jul 31, 2020

Consider the following script that runs fine on my PC (with no output):

for lname in ['LZMA', 'BZ2', 'Gzip']:
    lib = __import__(lname.lower())
    cls = getattr(lib, lname + 'File')
    #  cls = getattr(lib, 'open')  # uncomment to see that this also works
    example_payload = b'a'
    with cls('a.bin', 'w') as f:
        f.write(example_payload)
    with cls('a.bin', 'r') as f:
        if f.read() != example_payload:
            print('WTF!')

All three compression libraries in the Python standard library follow a pattern: there's an SomethingFile object that supports reading and writing just as if it was a regular file. Perhaps it makes sense to add such a class to python-zstandard as well?

@d33tah d33tah changed the title Add ZstandardFile class that supports 'r' and 'w', just like in Python's zstd Add ZstandardFile class that supports 'r' and 'w', just like in Python's stdlib Jul 31, 2020
@d33tah d33tah changed the title Add ZstandardFile class that supports 'r' and 'w', just like in Python's stdlib Add ZstandardFile class (and "open" method) that supports 'r' and 'w', just like in Python's stdlib Jul 31, 2020
@d33tah d33tah changed the title Add ZstandardFile class (and "open" method) that supports 'r' and 'w', just like in Python's stdlib Add ZstandardFile class (and "open" method) that supports 'r' and 'w' modes, just like in Python's stdlib Jul 31, 2020
@d33tah
Copy link
Author

d33tah commented Jul 31, 2020

Here's my lazy attempt at addressing this:

import zstandard
import contextlib

old_open = open

@contextlib.contextmanager
def open(*, fileobj=None, file=None, mode='r'):
    if mode not in ('r', 'w'):
        raise ValueError('Mode should be "r" or "w")')
    if not fileobj and not file:
        raise ValueError('either fileobj or file must be provided')
    if mode == "r":
        cctx = zstandard.ZstdDecompressor()
        if file:
            with old_open(file, 'rb') as fh, cctx.stream_reader(fh) as reader:
                yield reader
        else:
            yield cctx.stream_reader(fileobj)
    elif mode == "w":
        cctx = zstandard.ZstdCompressor()
        if file:
            with old_open(file, 'wb') as fh, cctx.stream_writer(fh) as writer:
                yield writer
        else:
            yield cctx.stream_writer(fileobj)

@indygreg
Copy link
Owner

I'm closing this issue because it is a duplicate of #23 and #64.

I do agree that having these APIs would be a good idea. I may add these to the next release. We'll see.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants