ContentDecodingError on PyPy with 6+

[Gallaecio]

With vcrpy >= 6.0.0, some tests in https://github.com/scrapinghub/python-scrapinghub started failing on PyPy.

If you clone that repo and run:

tox -e pypy310-json -- tests/client/test_activity.py::test_activity_iter

The test fails with requests.exceptions.ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check')).

$ tox -e pypy310-json -- tests/client/test_activity.py::test_activity_iter
pypy310-json: recreate env because requirements removed: vcrpy<6
pypy310-json: remove tox env folder /home/adrian/temporal/python-scrapinghub/.tox/pypy310-json
pypy310-json: install_deps> python -I -m pip install -r /home/adrian/temporal/python-scrapinghub/requirements-base.txt -r /home/adrian/temporal/python-scrapinghub/requirements-test.txt
.pkg: _optional_hooks> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
pypy310-json: install_package_deps> python -I -m pip install 'requests>=1.0' 'retrying>=1.3.3' 'six>=1.10.0'
pypy310-json: install_package> python -I -m pip install --force-reinstall --no-deps /home/adrian/temporal/python-scrapinghub/.tox/.tmp/package/24/scrapinghub-2.4.0.tar.gz
pypy310-json: commands[0]> py.test --cov=scrapinghub --cov-report=xml tests/client/test_activity.py::test_activity_iter
=================================================================================================== test session starts ====================================================================================================
platform linux -- Python 3.10.13[pypy-7.3.13-final], pytest-8.3.4, pluggy-1.5.0
cachedir: .tox/pypy310-json/.pytest_cache
rootdir: /home/adrian/temporal/python-scrapinghub
configfile: pytest.ini
plugins: cov-6.0.0
collected 1 item                                                                                                                                                                                                           

tests/client/test_activity.py F                                                                                                                                                                                      [100%]

========================================================================================================= FAILURES =========================================================================================================
____________________________________________________________________________________________________ test_activity_iter ____________________________________________________________________________________________________

self = <urllib3.response.HTTPResponse object at 0x00007080f621a3d8>
data = b'{"event":"job:cancelled","job":"2222222/2/3","user":"john"}\n{"event":"job:completed","job":"2222222/2/3","user":"jo...222/1/192","user":"admin"}\n{"event":"job:deleted","job":"2222222/1/193","user":"admin"}\n{"event":"job:deleted","job"'
decode_content = True, flush_decoder = False

    def _decode(self, data, decode_content, flush_decoder):
        """
        Decode the data passed in and potentially flush the decoder.
        """
        if not decode_content:
            return data
    
        try:
            if self._decoder:
>               data = self._decoder.decompress(data)

.tox/pypy310-json/lib/pypy3.10/site-packages/urllib3/response.py:407: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib3.response.GzipDecoder object at 0x00007080f621a480>
data = b'{"event":"job:cancelled","job":"2222222/2/3","user":"john"}\n{"event":"job:completed","job":"2222222/2/3","user":"jo...222/1/192","user":"admin"}\n{"event":"job:deleted","job":"2222222/1/193","user":"admin"}\n{"event":"job:deleted","job"'

    def decompress(self, data):
        ret = bytearray()
        if self._state == GzipDecoderState.SWALLOW_DATA or not data:
            return bytes(ret)
        while True:
            try:
>               ret += self._obj.decompress(data)
E               zlib.error: Error -3 while decompressing data: incorrect header check

.tox/pypy310-json/lib/pypy3.10/site-packages/urllib3/response.py:94: error

During handling of the above exception, another exception occurred:

    def generate():
        # Special case for urllib3.
        if hasattr(self.raw, "stream"):
            try:
>               yield from self.raw.stream(chunk_size, decode_content=True)

.tox/pypy310-json/lib/pypy3.10/site-packages/requests/models.py:820: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/pypy310-json/lib/pypy3.10/site-packages/urllib3/response.py:628: in stream
    data = self.read(amt=amt, decode_content=decode_content)
.tox/pypy310-json/lib/pypy3.10/site-packages/urllib3/response.py:600: in read
    data = self._decode(data, decode_content, flush_decoder)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib3.response.HTTPResponse object at 0x00007080f621a3d8>
data = b'{"event":"job:cancelled","job":"2222222/2/3","user":"john"}\n{"event":"job:completed","job":"2222222/2/3","user":"jo...222/1/192","user":"admin"}\n{"event":"job:deleted","job":"2222222/1/193","user":"admin"}\n{"event":"job:deleted","job"'
decode_content = True, flush_decoder = False

    def _decode(self, data, decode_content, flush_decoder):
        """
        Decode the data passed in and potentially flush the decoder.
        """
        if not decode_content:
            return data
    
        try:
            if self._decoder:
                data = self._decoder.decompress(data)
        except self.DECODER_ERROR_CLASSES as e:
            content_encoding = self.headers.get("content-encoding", "").lower()
>           raise DecodeError(
                "Received response with content-encoding: %s, but "
                "failed to decode it." % content_encoding,
                e,
            )
E           urllib3.exceptions.DecodeError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))

.tox/pypy310-json/lib/pypy3.10/site-packages/urllib3/response.py:410: DecodeError

During handling of the above exception, another exception occurred:

project = <scrapinghub.client.projects.Project object at 0x00007080f4359910>

    def test_activity_iter(project):
        _add_test_activity(project)
>       activity = project.activity.iter()

tests/client/test_activity.py:24: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
scrapinghub/client/activity.py:56: in iter
    return self._origin.list(**params)
scrapinghub/hubstorage/activity.py:11: in list
    return self.apiget(params=params)
scrapinghub/hubstorage/resourcetype.py:86: in apiget
    return self.apirequest(_path, method='GET', **kwargs)
scrapinghub/hubstorage/resourcetype.py:79: in apirequest
    return jldecode(self._iter_lines(_path, **kwargs))
scrapinghub/hubstorage/resourcetype.py:68: in _iter_lines
    r = self.client.request(**kwargs)
scrapinghub/client/exceptions.py:69: in wrapped
    return method(*args, **kwargs)
scrapinghub/client/__init__.py:26: in request
    return super(HubstorageClient, self).request(*args, **kwargs)
scrapinghub/hubstorage/client.py:116: in request
    return self.retrier.call(invoke_request)
.tox/pypy310-json/lib/pypy3.10/site-packages/retrying.py:257: in call
    return attempt.get(self._wrap_exception)
.tox/pypy310-json/lib/pypy3.10/site-packages/retrying.py:301: in get
    six.reraise(self.value[0], self.value[1], self.value[2])
.tox/pypy310-json/lib/pypy3.10/site-packages/six.py:724: in reraise
    raise value
.tox/pypy310-json/lib/pypy3.10/site-packages/retrying.py:251: in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
scrapinghub/hubstorage/client.py:106: in invoke_request
    r = self.session.request(**kwargs)
.tox/pypy310-json/lib/pypy3.10/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
.tox/pypy310-json/lib/pypy3.10/site-packages/requests/sessions.py:746: in send
    r.content
.tox/pypy310-json/lib/pypy3.10/site-packages/requests/models.py:902: in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def generate():
        # Special case for urllib3.
        if hasattr(self.raw, "stream"):
            try:
                yield from self.raw.stream(chunk_size, decode_content=True)
            except ProtocolError as e:
                raise ChunkedEncodingError(e)
            except DecodeError as e:
>               raise ContentDecodingError(e)
E               requests.exceptions.ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))

.tox/pypy310-json/lib/pypy3.10/site-packages/requests/models.py:824: ContentDecodingError
--------------------------------------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------------------------------------
WARNING  HubstorageClient:client.py:94 Messagepack is not available, please ensure that msgpack-python library is properly installed.

---------- coverage: platform linux, python 3.10.13-final-0 ----------
Coverage XML written to file coverage.xml

================================================================================================= short test summary info ==================================================================================================
FAILED tests/client/test_activity.py::test_activity_iter - requests.exceptions.ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))
==================================================================================================== 1 failed in 1.70s =====================================================================================================
pypy310-json: exit 1 (3.13 seconds) /home/adrian/temporal/python-scrapinghub> py.test --cov=scrapinghub --cov-report=xml tests/client/test_activity.py::test_activity_iter pid=3127325
  pypy310-json: FAIL code 1 (14.26=setup[11.13]+cmd[3.13] seconds)
  evaluation failed :( (14.31 seconds)

But if you set vcrpy<6 in requirements-test.txt and re-run the same command, it passes.

Please, let me know if I should provide a minimal reproducible example, or if this is some expected change in 6+ and I need to update the repo somehow to adapt to it.