Mamba fails to extract the latest package on macos #35

isuruf · 2023-07-05T06:01:55Z

Mamba fails with

critical libmamba Pathname can't be converted from UTF-8 to current locale.
error    libmamba Error opening for reading "/Users/runner/miniforge3/pkgs/python-libarchive-c-5.0-py310h2ec42d9_0/info/index.json": No such file or directory
error    libmamba Error when extracting package: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal

The contents of the package include

$ ls -1 pkgs/python-libarchive-c-5.0-py311h6eed73b_0/info/test/tests/data/
flags.tar
special.tar
tar_relative.tar
testtar.README
testtar.tar
testtar.tar.json
unicode2.zip
unicode2.zip.json
unicode.tar
unicode.tar.json
unicode.zip
unicode.zip.json
프로그램.README
프로그램.zip
프로그램.zip.json

The text was updated successfully, but these errors were encountered:

hmaarrfk · 2023-07-05T06:26:24Z

repodata patches recipe seems to be ok at generating the diff. so it isn't in the metadata.

hmaarrfk · 2023-07-05T06:26:45Z

 ~/git/feedstocks/conda-forge-repodata-patches-feedstock/recipe [main|…2]
mark $ python show_diff.py
Downloading: https://conda.anaconda.org/conda-forge/noarch/repodata_from_packages.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/noarch/repodata.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/linux-64/repodata_from_packages.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/linux-64/repodata.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/linux-armv7l/repodata_from_packages.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/linux-armv7l/repodata.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/linux-aarch64/repodata_from_packages.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/linux-aarch64/repodata.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/linux-ppc64le/repodata_from_packages.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/linux-ppc64le/repodata.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/osx-64/repodata_from_packages.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/osx-64/repodata.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/osx-arm64/repodata_from_packages.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/osx-arm64/repodata.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/win-32/repodata_from_packages.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/win-32/repodata.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/win-64/repodata_from_packages.json.bz2
Downloading: https://conda.anaconda.org/conda-forge/win-64/repodata.json.bz2

isuruf · 2023-07-05T06:48:18Z

It's not the metadata. It's the contents of the package. There are some files with unicode characters.

wolfv · 2023-07-05T07:01:14Z

I'll have a look if we can fix this in mamba

cxong · 2023-07-05T07:11:29Z

meanwhile, pinning python-libarchive-c to version 4.0 is a workaround

vnlitvinov · 2023-07-05T08:40:25Z

@cxong

meanwhile, pinning python-libarchive-c to version 4.0 is a workaround

How do I do that in a feedstock recipe? As far as I can tell, it's installed even before my meta.yml comes in play by the CF tools when installing conda-build.

jaimergp · 2023-07-05T09:57:34Z

Should we temporarily mark those as broken til the mamba fix is deployed?

jaimergp · 2023-07-05T10:13:20Z

See conda-forge/admin-requests#750

duncanmmacleod · 2023-07-05T10:40:03Z

For the record, I also see this on linux-64 when running in a docker container powered by gitlab-ci:

$ conda list --name base
# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
brotli                    1.0.9                h166bdaf_8    conda-forge
brotli-bin                1.0.9                h166bdaf_8    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.5.7             hbcca054_0    conda-forge
certifi                   2023.5.7           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.1.0          py310hff52083_0    conda-forge
conda-package-handling    2.0.2              pyh38be061_0    conda-forge
conda-package-streaming   0.8.0              pyhd8ed1ab_0    conda-forge
cryptography              41.0.1          py310h75e40e8_0    conda-forge
fmt                       9.1.0                h924138e_0    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libarchive                3.6.2                h039dbb9_1    conda-forge
libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
libbrotlidec              1.0.9                h166bdaf_8    conda-forge
libbrotlienc              1.0.9                h166bdaf_8    conda-forge
libcurl                   8.1.2                h409715c_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libmamba                  1.4.6                h658169a_0    conda-forge
libmambapy                1.4.6           py310h8aae740_0    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsolv                   0.7.24               hfc55251_1    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxml2                   2.11.4               h0d562d8_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.4.6           py310h51d5547_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
openssl                   3.1.1                hd590300_1    conda-forge
pip                       23.1.2             pyhd8ed1ab_0    conda-forge
pluggy                    1.2.0              pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.4           py310h5764c6d_1    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.12         hd12c33a_0_cpython    conda-forge
python_abi                3.10                    3_cp310    conda-forge
readline                  8.2                  h8228510_1    conda-forge
reproc                    14.2.4               h0b41bf4_0    conda-forge
reproc-cpp                14.2.4               hcb278e6_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.17.32         py310h2372a71_0    conda-forge
ruamel.yaml.clib          0.2.7           py310h1fa729e_1    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tqdm                      4.65.0             pyhd8ed1ab_1    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   2.0.3              pyhd8ed1ab_0    conda-forge
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml-cpp                  0.7.0                h27087fc_2    conda-forge
zstandard                 0.19.0          py310h1275a96_2    conda-forge
zstd                      1.5.2                h3eb15da_6    conda-forge
$ retry() { # collapsed multi-line command
$ retry mamba create --name render --quiet --yes --file requirements.txt
critical libmamba Pathname can't be converted from UTF-8 to current locale.
error    libmamba Error opening for reading "/builds/computing/conda/.cache/conda/pkgs/python-libarchive-c-5.0-py311h38be061_0/info/index.json": No such file or directory
error    libmamba Error when extracting package: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
Found incorrect download: python-libarchive-c. Aborting
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1124, in __call__
        return func(*args, **kwargs)
      File "/opt/conda/lib/python3.10/site-packages/mamba/mamba.py", line 924, in exception_converter
        raise e
      File "/opt/conda/lib/python3.10/site-packages/mamba/mamba.py", line 917, in exception_converter
        exit_code = _wrapped_main(*args, **kwargs)
      File "/opt/conda/lib/python3.10/site-packages/mamba/mamba.py", line 875, in _wrapped_main
        result = do_call(parsed_args, p)
      File "/opt/conda/lib/python3.10/site-packages/mamba/mamba.py", line 750, in do_call
        exit_code = create(args, parser)
      File "/opt/conda/lib/python3.10/site-packages/mamba/mamba.py", line 599, in create
        return install(args, parser, "create")
      File "/opt/conda/lib/python3.10/site-packages/mamba/mamba.py", line 555, in install
        transaction.fetch_extract_packages()
    RuntimeError: Found incorrect download: python-libarchive-c. Aborting
`$ /opt/conda/bin/mamba create --name render --quiet --yes --file requirements.txt`
  environment variables:
                 CIO_TEST=<not set>
           CI_CONFIG_PATH=.gitlab-ci.yml
CI_MERGE_REQUEST_PROJECT_PATH=computing/conda
CI_MERGE_REQUEST_REF_PATH=refs/merge-requests/1137/head
CI_MERGE_REQUEST_SOURCE_PROJECT_PATH=duncanmmacleod/conda
          CI_PROJECT_PATH=computing/conda
                  CONDARC=/builds/computing/conda/.condarc
           CONDA_BLD_PATH=/builds/computing/conda/conda-bld
                CONDA_DIR=/opt/conda
          CONDA_ENVS_PATH=/builds/computing/conda/envs
                CONDA_EXE=/opt/conda/bin/conda
              CONDA_FORGE=true
          CONDA_PKGS_DIRS=/builds/computing/conda/.cache/conda/pkgs
         CONDA_PYTHON_EXE=/opt/conda/bin/python
               CONDA_ROOT=/opt/conda
              CONDA_SHLVL=0
           CURL_CA_BUNDLE=<not set>
               CVMFS_PATH=/cvmfs/software.igwn.org/conda
               LD_PRELOAD=<not set>
                     PATH=/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr
                          /sbin:/usr/bin:/sbin:/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
     active environment : None
            shell level : 0
       user config file : /root/.condarc
 populated config files : /opt/conda/.condarc
                          /builds/computing/conda/.condarc
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.31=0
                          __linux=4.18.0=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/igwn/linux-64
                          https://conda.anaconda.org/igwn/noarch
          package cache : /builds/computing/conda/.cache/conda/pkgs
       envs directories : /builds/computing/conda/envs
                          /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/23.1.0 requests/2.31.0 CPython/3.10.12 Linux/4.18.0-425.10.1.el8_7.x86_64 ubuntu/20.04.6 glibc/2.31
                UID:GID : 0:0
             netrc file : None
           offline mode : False
An unexpected error has occurred. Conda has prepared the above report.

The image is igwn/base:conda which is built FROM condaforge/mambaforge.

jpfeuffer · 2023-07-05T11:16:15Z

Same problem in conda-forge/staged-recipes#23023
and on bioconda.

wolfv · 2023-07-05T11:21:24Z

Argh, sorry about this.

I have debugged a bit and it seems to originate in libarchive, specifically this code:

            r = archive_read_next_header(a, &entry);
            if (r == ARCHIVE_EOF)
            {
                break;
            }
            if (r < ARCHIVE_OK)
            {
                LOG_INFO << "Writing not OK! " << archive_error_string(a);
                throw std::runtime_error(archive_error_string(a));
            }

logs for me:

info     libmamba Extracitng to /Users/wolfv/Programs/mamba/cbuild/ ... 
info     libmamba starting the writing loop
info     libmamba writing not OK! Pathname can't be converted from UTF-8 to current locale.
critical libmamba Pathname can't be converted from UTF-8 to current locale.

I am trying to see what our options are - wether we need a new libarchive build with additional UTF8 handling or if we need to set some things differently in libmamba.

bergtholdt · 2023-07-05T12:45:19Z

Can also confirm for linux-64 latest Mambaforge, trying to install Boa.

MRedies · 2023-07-05T13:21:57Z

I also get it for linux-64. Can you mark these as broken as well?

wolfv · 2023-07-05T16:02:41Z

We identified the source of the issue and have a fix (in mamba): mamba-org/mamba#2655

Should be released tonight or tomorrow morning!

jaimergp · 2023-07-05T16:16:09Z

I also get it for linux-64. Can you mark these as broken as well?

See conda-forge/admin-requests#751

JohanMabille · 2023-07-06T07:28:32Z

Mamba 1.4.7 has been released with a fix to this issue. It should be available on conda-forge soon.

isuruf · 2023-07-06T19:44:06Z

Shall we delete those test files for the package @conda-forge/python-libarchive-c? Even though this is fixed in mamba 1.4.7, it doesn't work with older versions and since it's a pretty important package, I think we shouldn't release it where it has the potential to not work.

kuepe-sl · 2023-07-07T06:38:38Z

I'm using boa to build packages. Boa currently is version-pinned to Mamba <=1.4.2, so I need python-libarchive-c to work with older Mamba versions.

JohanMabille · 2023-07-07T06:48:46Z

Shall we delete those test files for the package @conda-forge/python-libarchive-c? Even though this is fixed in mamba 1.4.7, it doesn't work with older versions and since it's a pretty important package, I think we shouldn't release it where it has the potential to not work.

I think so too.

jaimergp · 2023-07-07T09:13:43Z

Opened #36

isuruf mentioned this issue Jul 5, 2023

All macOS builds are failing conda-forge/status#149

Closed

zklaus mentioned this issue Jul 5, 2023

Improve requirements conda-forge/libnetcdf-feedstock#178

Merged

5 tasks

jaimergp mentioned this issue Jul 5, 2023

mark python-libarchive-c 5.0 as broken on osx conda-forge/admin-requests#750

Merged

9 tasks

duncanmmacleod mentioned this issue Jul 5, 2023

Add arby as runtime requirement conda-forge/cwinpy-feedstock#40

Merged

4 tasks

h-vetinari mentioned this issue Jul 5, 2023

[bot-automerge] pysr v0.14.3 conda-forge/pysr-feedstock#96

Merged

3 tasks

timosachsenberg mentioned this issue Jul 5, 2023

Update pyopenms recipe to fix zlib issue bioconda/bioconda-recipes#41716

Merged

jaimergp mentioned this issue Jul 7, 2023

Remove unicode-named test files #36

Merged

5 tasks

k-dominik mentioned this issue Jul 11, 2023

unfreeze cache before exporting data from tda workflow ilastik/ilastik#2723

Merged

This was referenced Jul 1, 2024

Read the docs build failures nextstrain/docs.nextstrain.org#209

Closed

Latest Sphinx (7.3.7) corrupted install on mambaforge readthedocs/readthedocs.org#11416

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mamba fails to extract the latest package on macos #35

Mamba fails to extract the latest package on macos #35

isuruf commented Jul 5, 2023

hmaarrfk commented Jul 5, 2023

hmaarrfk commented Jul 5, 2023

isuruf commented Jul 5, 2023

wolfv commented Jul 5, 2023

cxong commented Jul 5, 2023

vnlitvinov commented Jul 5, 2023

jaimergp commented Jul 5, 2023

jaimergp commented Jul 5, 2023

duncanmmacleod commented Jul 5, 2023

jpfeuffer commented Jul 5, 2023

wolfv commented Jul 5, 2023

bergtholdt commented Jul 5, 2023

MRedies commented Jul 5, 2023

wolfv commented Jul 5, 2023 •

edited

Loading

jaimergp commented Jul 5, 2023

JohanMabille commented Jul 6, 2023

isuruf commented Jul 6, 2023

kuepe-sl commented Jul 7, 2023

JohanMabille commented Jul 7, 2023 •

edited

Loading

jaimergp commented Jul 7, 2023

Mamba fails to extract the latest package on macos #35

Mamba fails to extract the latest package on macos #35

Comments

isuruf commented Jul 5, 2023

hmaarrfk commented Jul 5, 2023

hmaarrfk commented Jul 5, 2023

isuruf commented Jul 5, 2023

wolfv commented Jul 5, 2023

cxong commented Jul 5, 2023

vnlitvinov commented Jul 5, 2023

jaimergp commented Jul 5, 2023

jaimergp commented Jul 5, 2023

duncanmmacleod commented Jul 5, 2023

jpfeuffer commented Jul 5, 2023

wolfv commented Jul 5, 2023

bergtholdt commented Jul 5, 2023

MRedies commented Jul 5, 2023

wolfv commented Jul 5, 2023 • edited Loading

jaimergp commented Jul 5, 2023

JohanMabille commented Jul 6, 2023

isuruf commented Jul 6, 2023

kuepe-sl commented Jul 7, 2023

JohanMabille commented Jul 7, 2023 • edited Loading

jaimergp commented Jul 7, 2023

wolfv commented Jul 5, 2023 •

edited

Loading

JohanMabille commented Jul 7, 2023 •

edited

Loading