repodata_use_zst causes Invalid cross-device link error

[nathanweeks]

Troubleshooting docs

• My problem is not solved in the Troubleshooting docs

Search tried in issue tracker

repodata_use_zst

Latest version of Mamba

• My problem is not solved with the latest version

Tried in Conda?

Not applicable

Describe your issue

On a host where /tmp and $HOME are on different partitions, micromamba create -n <env>... results in an Invalid cross-device link error.

mamba info / micromamba info

environment : None (not found)
           env location : -
      user config files : /home/user/.mambarc
 populated config files : /home/user/.condarc
       libmamba version : 1.2.0
     micromamba version : 1.2.0
           curl version : libcurl/7.87.0 OpenSSL/1.1.1s zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0
     libarchive version : libarchive 3.6.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.2
       virtual packages : __unix=0=0
                          __linux=3.10.0=0
                          __glibc=2.17=0
                          __archspec=1=x86_64
               channels : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
       base environment : /home/user/micromamba
               platform : linux-64

Logs

$ micromamba create -v -n samtools samtools

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

warning  libmamba 'root_prefix' set with default value: /home/user/micromamba
info     libmamba You have not set the 'root_prefix' environment variable.
    To permanently modify the root prefix location, either:
    - set the 'MAMBA_ROOT_PREFIX' environment variable
    - use the '-r,--root-prefix' CLI option
    - use 'micromamba shell init ...' to initialize your shell
        (then restart or source the contents of the shell init script)
info     libmamba Parsing MatchSpec samtools
info     libmamba Searching index cache file for repo 'https://conda.anaconda.org/conda-forge/linux-64/repodata.json'
conda-forge/linux-64                                        Using cache
info     libmamba Searching index cache file for repo 'https://conda.anaconda.org/conda-forge/noarch/repodata.json'
conda-forge/noarch                                          Using cache
info     libmamba Searching index cache file for repo 'https://conda.anaconda.org/bioconda/linux-64/repodata.json'
bioconda/linux-64                                           Using cache
info     libmamba Searching index cache file for repo 'https://conda.anaconda.org/bioconda/noarch/repodata.json'
bioconda/noarch                                             Using cache
info     libmamba Searching index cache file for repo 'https://repo.anaconda.com/pkgs/main/linux-64/repodata.json'
info     libmamba No valid cache found
info     libmamba Expired cache (or invalid mod/etag headers) found at '/home/user/micromamba/pkgs'
info     libmamba Using OpenSSL backend
info     libmamba Searching index cache file for repo 'https://repo.anaconda.com/pkgs/main/noarch/repodata.json'
info     libmamba No valid cache found
info     libmamba Expired cache (or invalid mod/etag headers) found at '/home/user/micromamba/pkgs'
info     libmamba Searching index cache file for repo 'https://repo.anaconda.com/pkgs/r/linux-64/repodata.json'
info     libmamba No valid cache found
info     libmamba Expired cache (or invalid mod/etag headers) found at '/home/user/micromamba/pkgs'
info     libmamba Searching index cache file for repo 'https://repo.anaconda.com/pkgs/r/noarch/repodata.json'
info     libmamba No valid cache found
info     libmamba Expired cache (or invalid mod/etag headers) found at '/home/user/micromamba/pkgs'
info     libmamba Starting to download targets
[+] 0.2s
pkgs/main/linux-64 (check zst) ━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s Checking  0.2s
pkgs/main/noarch (check zst)   ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s Checking  0.2sinfo     libmamba Transfer done for 'pkgs/main/noarch (check zst)'
info     libmamba Transfer finalized, status: 200 [https://repo.anaconda.com/pkgs/main/noarch/repodata.json.zst] 0 bytes
info     libmamba Checked: https://repo.anaconda.com/pkgs/main/noarch/repodata.json.zst [200]
info     libmamba Transfer done for 'pkgs/main/linux-64 (check zst)'
pkgs/main/noarch (check zst)                        Checked  0.3s
pkgs/main/linux-64 (check zst)                      Checked  0.3s
info     libmamba Starting to download targets
[+] 0.0s
info     libmamba Transfer done for 'pkgs/main/noarch'
info     libmamba Reading cache files '/home/user/micromamba/pkgs/cache/497deca9.solv.*' for repo index 'https://conda.anaconda.org/conda-forge/linux-64'
info     libmamba Attempt load from solv "/home/user/micromamba/pkgs/cache/497deca9.solv"
info     libmamba Metadata solv file: https://conda.anaconda.org/conda-forge/linux-64/repodata.json 1 W/"b974d0059674378f6e80cd2990d323c2" Fri, 20 Jan 2023 20:19:27 GMT 1.1_0.7.23
info     libmamba Metadata from SOLV are valid
info     libmamba Reading cache files '/home/user/micromamba/pkgs/cache/09cdf8bf.solv.*' for repo index 'https://conda.anaconda.org/conda-forge/noarch'
info     libmamba Attempt load from solv "/home/user/micromamba/pkgs/cache/09cdf8bf.solv"
info     libmamba Metadata solv file: https://conda.anaconda.org/conda-forge/noarch/repodata.json 1 W/"f83c20b4c298d5df7fa07b0ed90f65eb" Fri, 20 Jan 2023 20:46:59 GMT 1.1_0.7.23
info     libmamba Metadata from SOLV are valid
info     libmamba Reading cache files '/home/user/micromamba/pkgs/cache/ffeee55f.solv.*' for repo index 'https://conda.anaconda.org/bioconda/linux-64'
info     libmamba Attempt load from solv "/home/user/micromamba/pkgs/cache/ffeee55f.solv"
info     libmamba Metadata solv file: https://conda.anaconda.org/bioconda/linux-64/repodata.json 1 W/"97953b33aa1ad17678b09e5ebe179a40-4" Fri, 20 Jan 2023 14:07:35 GMT 1.1_0.7.23
info     libmamba Metadata from SOLV are valid
info     libmamba Reading cache files '/home/user/micromamba/pkgs/cache/2a957770.solv.*' for repo index 'https://conda.anaconda.org/bioconda/noarch'
info     libmamba Attempt load from solv "/home/user/micromamba/pkgs/cache/2a957770.solv"
info     libmamba Metadata solv file: https://conda.anaconda.org/bioconda/noarch/repodata.json 1 W/"3fb68e720487a5de0b44ff6bd2bdc7ab-4" Fri, 20 Jan 2023 17:08:53 GMT 1.1_0.7.23
info     libmamba Metadata from SOLV are valid
info     libmamba Freeing pool.
critical libmamba Multiple errors occured:
    filesystem error: cannot rename: Invalid cross-device link [/tmp/mambafvc7igVHn7I] [/home/user/micromamba/pkgs/cache/3e39a7aa.json]
    Subdir pkgs/main/noarch not loaded!
    Subdir pkgs/r/noarch not loaded!

environment.yml

No response

~/.condarc

channels:
  - conda-forge
  - bioconda
  - defaults
channel_priority: strict
repodata_use_zst: true

[hadim]

I confirm the error on my system.

[jonashaag]

My guess is that we’re using an atomic rename that’s not supported across file systems

[jonashaag]

Can you please test #2242 once it's built (use the binaries from the Static Builds pipeline)

[hadim]

The static build at https://github.com/mamba-org/mamba/suites/10512667339/artifacts/521981998 seems to be working for me (and still fail with 1.2). I have tested after a clean --all.

[nathanweeks]

The static build at https://github.com/mamba-org/mamba/suites/10512667339/artifacts/521981998 seems to be working for me (and still fail with 1.2). I have tested after a clean --all.

It works in my environment as well.

[wolfv]

Thanks for debugging and thanks for the fix @jonashaag

I am wondering if we should do it like this, or if we should instead try to create the temporary file "next" to the final file so that we can rename it without cross-device issues...

[jonashaag]

Sure that would also be an option! Although it still might be the case that you can’t use fs::rename because even on the same filesystem, atomic renames might be unavailable (eg network filesystem)

[romain-intel]

As a quick question on this functionality: I ran into this issue for micromamba and then tried just regular mamba but for some reason it does not download the zst files and sticks with the older version. Is there something else I need to do? I did set repodata_use_zst: true in all .condarc and .mambarc files I could find (and even created new ones just to be safe). I'm probably missing something dreadfully obvious.

Thanks!