InvalidArchiveError raised for valid tarballs

[desilinguist]

For some reason, some of my large conda packages are no longer installed properly and fail with InvalidArchiveError. I have narrowed down the problem to conda-package-handling. Here's what I see:

>>> from conda_package_handling.api import extract, libarchive_enabled
>>> libarchive_enabled
True
>>> extract('/tmp/gug-data-1.0-0.tar.bz2')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python/conda_default/lib/python3.6/site-packages/conda_package_handling/api.py", line 58, in extract
    SUPPORTED_EXTENSIONS[ext].extract(fn, dest_dir, components=components)
  File "/opt/python/conda_default/lib/python3.6/site-packages/conda_package_handling/tarball.py", line 135, in extract
    _tar_xf(fn, dest_dir)
  File "/opt/python/conda_default/lib/python3.6/site-packages/conda_package_handling/tarball.py", line 88, in _tar_xf
    archive_utils.extract_file(tarball)
  File "/opt/python/conda_default/lib/python3.6/site-packages/conda_package_handling/archive_utils.py", line 15, in extract_file
    raise InvalidArchiveError(tarball, error_str.decode('utf-8'))
conda_package_handling.exceptions.InvalidArchiveError: Error with archive /tmp/gug-data-1.0-0.tar.bz2.  You probably need to delete and re-download or re-create this file.  Message from libarchive was:

Seek failed

This happens on a RHEL 7.9 machine with conda-package-handling v1.6.1. However, I am also able to reproduce the same issue on my MacBook Pro (macOS 10.14.6) with conda-package-handling v1.7.0. If I manually delete site-packages/conda_package_handling/archive_utils.py and site-packages/conda_package_handling/archive_utils.XXX.so from my conda environment to make it backoff to tarfile, the above snippet works fine without any errors.

>>> from conda_package_handling.api import extract, libarchive_enabled
>>> libarchive_enabled
False
>>> extract('/tmp/gug-data-1.0-0.tar.bz2')
>>> 

Here are the contents of gug-data-1.0-0.tar.bz2:

        84 Feb  4  2015 info/files
       140 Dec 31  1969 info/index.json
       482 Feb  4  2015 info/recipe.json
       232 Feb  4  2015 info/recipe/build.sh
       314 Feb  4  2015 info/recipe/meta.yaml
        68 Feb  4  2015 info/recipe/meta.yaml.orig
2241453926 Feb  4  2015 share/giga3gram.lm.binary
2145097736 Feb  4  2015 share/ngrams_gigaword_afp_apw_cna_ltw_nyt_xin.marisa_trie

Any ideas what could be happening? The same conda package was being installed just fine a few days ago.

[desilinguist]

Interesting, it looks like downgrading to v1.6.0 works as well. So some change was introduced starting in 1.6.1 that broke something.

[desilinguist]

Bump. Any updates on this?

[nehaljwani]

@desilinguist Could you attach a zip of gug-data-1.0-0.tar.bz2 to this issue?

[desilinguist]

Hi @nehaljwani, that file is about 3G in size. Do you really want me to attach it here?

[nehaljwani]

If you could provide any tarball to reproduce the issue, it would help!

[desilinguist]

Hi @nehaljwani, I uploaded the tarball to my Google Drive. Here's the link. Please let me know once you have downloaded it so I can remove it from my drive. Thanks!

[desilinguist]

Hi @nehaljwani, please confirm that you are able to access the file so I can remove it.

[desilinguist]

Any progress on this issue? @nehaljwani

[seemethere]

We're also currently experiencing this with nightlies from pytorch.

You can access them for testing with:

curl -fL -O https://9581561-65600975-gh.circle-artifacts.com/0/final_pkgs/pytorch-1.8.0.dev20201214-py3.6_cuda11.0.221_cudnn8.0.5_0.tar.bz2

[mattip]

The stacktrace seems to point to something in the c-extension module for archive handling.

Digging around a bit, I see that the 1.7.0 tag mentions "Updating libarchive-static to 3.4.2", but I could not find mention of this in the changelog, which seems to skip 1.7.0 entirely. Maybe that was a typo, since the conda recipe seems to link to libarchive-static 3.3.0. Another change to the archive handling in the past 12 months was the fix to #61, which added checks to strdup.

[mattip]

When you downgrade conda-package-handling to 1.6.0, does it also downgrade any other packages?

[desilinguist]

@mattip I don’t think so but I use the --force flag when downgrading to be on the safe side.

[desilinguist]

Is this issue being investigated at all?

[rgommers]

@desilinguist yes, see the PR from @leej3 linked right above this issue.

[desilinguist]

Oh, that's great! Sorry I totally missed that. Looking forward to the new release with the fixes. I am glad that the tarball I provided seems to have been useful in fixing the issue.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!