conda installation broken again with error installing cachetools-2.1.0-py_0

[shankari]

This appears to be a regression in the past few days. When I experimented with the dev builds for docker-compose (around e-mission/e-mission-docker@c66f1c8), everything worked. Then, when I started testing using Travis CI, around Thu Mar 12 23:15:25 2020), I started getting the error.

ERROR conda.core.link:_execute(507): An error occurred while installing package 'conda-forge::cachetools-2.1.0-py_0'.
FileNotFoundError(2, "No such file or directory: '/opt/conda/envs/testbug/bin/python3.6'")

There were no source code changes in this time, and no reason why this should have regressed. This is a conda issue, so I have filed conda/conda#9761

I am very disturbed about conda's dependency checking bitrotting, since the whole point of using conda was to avoid getting sucked into dependency hell.

[shankari]

This makes it fairly clear that it is time to upgrade the python dependencies, and I am not opposed to doing so, but I want to have reliable tests in place first so that I don't regress my own code.

[shankari]

This issue will track the temporary workarounds to get the tests to pass again before we upgrade the dependencies.

[shankari]

Using conda-tree (https://github.com/rvalieris/conda-tree), it turns out that cachetools is a dependency of https://google-auth.readthedocs.io/en/latest/

google-auth==1.2.1
  ├─ cachetools 2.1.0 [required: >=2.0.0,<3dev]
  │  └─ python 3.6.1 [required: any]
  │     └─ dependencies of python displayed above

[shankari]

Try 1: upgrade google-auth

version 1.7.0 updated the cachetools dependency

https://github.com/googleapis/google-auth-library-python/blob/a6d632923dfb9586f2d9d72f72640b2142277861/CHANGELOG.md#170

Still fails

Downloading and Extracting Packages
cachetools-3.1.1     | 11 KB     | ############################################# | 100%
google-auth-1.7.0    | 46 KB     | ############################################# | 100%
rsa-4.0              | 27 KB     | ############################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: failed
ERROR conda.core.link:_execute(507): An error occurred while installing package 'conda-forge::cachetools-3.1.1-py_0'.

Try 2: create an empty environment before installing all packages

since the secondary error is related to a FileNotFound for python3.6, let's try to install and ensure that it exists first.

still fails

Downloading and Extracting Packages
python-3.6.1         | 18.9 MB   | ############################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: failed
ERROR conda.core.link:_execute(507): An error occurred while installing package 'conda-forge::cachetools-3.1.1-py_0'.
FileNotFoundError(2, "No such file or directory: '/root/miniconda/envs/emission/bin/python3.6'")
Attempting to roll back.

Rolling back transaction: done

FileNotFoundError(2, "No such file or directory: '/root/miniconda/envs/emission/bin/python3.6'")

although the file clearly exists, is executable, and is not a symlink

# ls -al /root/miniconda/envs/emission/bin/python3.6
-rwxrwxr-x 1 root root 14040 May 11  2017 /root/miniconda/envs/emission/bin/python3.6

#  /root/miniconda/envs/emission/bin/python3.6
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:09:58)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Try 3: download the correct version from conda-forge manually and install it

That got rid of that error, but now the geojson package is failing

# curl -o /tmp/cachetools-2.1.0-py_0.tar.bz2 -L https://anaconda.org/conda-forge/cachetools/2.1.0/download/noarch/cachetools-2.1.0-py_0.tar.bz2

# conda install /tmp/cachetools-2.1.0-py_0.tar.bz2

Downloading and Extracting Packages
################################################################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done


# conda env update --name emission --file environment36.yml
Preparing transaction: done
Verifying transaction: done
Executing transaction: failed
ERROR conda.core.link:_execute(507): An error occurred while installing package 'conda-forge::geojson-2.3.0-py_0'.
FileNotFoundError(2, "No such file or directory: '/root/miniconda/envs/emission/bin/python3.6'")
Attempting to roll back.

Rolling back transaction: done

FileNotFoundError(2, "No such file or directory: '/root/miniconda/envs/emission/bin/python3.6'")

[shankari]

After going through and manually downloading every package that I was broken, I got this to work. The full list is:

• cachetools
• geojson (in environment file)
• jsonpickle (in environment file)
• more-itertools
• pyasn1
• pyasn1-modules

All the other modules, even from conda-forge, are installed correctly.
Let's check in this workaround so that we can get the tests to work again, and then we should really upgrade to the latest versions.

[shankari]

Fixed by #729