Conda reporting ResolvePackageNotFound on source installation

[kafitz]

When trying to install e-mission-server from source using source setup/setup.sh from the main branch, conda complains with ResolvePackageNotFound. Initially noticed on Debian 10, I just tried again with a fresh Ubuntu 18.04 (screenshot).

I create my conda environment like this with Python 3.6 to match the setup environments: conda create --name emission python=3.6

It seems odd since the packages like numpy==1.12.1=py36_0 seem to be listed here https://anaconda.org/anaconda/numpy/files?version=1.12.0 and the enabled channels are conda-forge and defaults. The download count on that page is also suspiciously low.

Any suggestions on my Conda usage? I've been working off the instructions here: https://e-mission.readthedocs.io/en/latest/install/manual_install/

[shankari]

I think that the problem is this

I create my conda environment like this with Python 3.6 to match the setup environments: conda create --name emission python=3.6

You don't need to create an emission environment. source setup/setup.sh should automatically do it for you.

If you still get the error, can you try ubuntu 16.04? I wonder if this is an issue with ubuntu 18.04. I just checked, and all the servers I am using seem to be on 16.04, including the docker image. It may be that conda doesn't support numpy==1.12.1=py36_0 on 18.04.

Note that the conda numpy and scipy packages are wrappers around c functions through cython, so the version of ubuntu/libc does matter.

[kafitz]

Receiving the same missing packages with a new install of Ubuntu 16.04 in a VM and running source setup/setup.sh after installing Anaconda. I also get this issue when I try to build the Dockerfile with docker build ..

[shankari]

I was able to reproduce this as well

$ source setup/setup.sh
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:
  - requests==2.14.2=py36_0
  - scipy==0.19.0=np112py36_0
  - numpy==1.12.1=py36_0
  - scikit-learn==0.18.1=np112py36_1

[shankari]

@kafitz you can use the example docker-compose configurations from
https://github.com/e-mission/e-mission-docker/
They use the image on dockerhub which should have all the packages installed correctly.

[shankari]

So I checked the github history, and the last update to the environment file is from Feb 2019.

commit 249010d1ed2ed16978bcdd01999eba8305085b7c
Author: K. Shankari <[email protected]>
Date:   Tue Feb 12 07:43:42 2019 -0800

    Set a particular python version again

commit 9b99263212875b2c2d9630782bdc57c503c7078b
Author: K. Shankari <[email protected]>
Date:   Mon Feb 11 22:44:57 2019 -0800

    Change the python version slightly + move tour model tests out

    Since they all import matplotlib, which we have now removed from the production
    environment

commit 50d31b6d923df8bd96c3fd0d79eaa5bd46cd9735
Author: K. Shankari <[email protected]>
Date:   Mon Feb 11 20:50:49 2019 -0800

So the file must just have bitrotted

[shankari]

So the file must just have bitrotted

This is extremely concerning since one of the big advantages of conda was supposed to be that you could pin the versions and have it work.

[shankari]

So at least for requests, conda clearly has the version

so why are we not getting it?

[shankari]

ok so I can also confirm that the package is not installable manually.

$ conda install requests==2.14.2=py36_0
Collecting package metadata (current_repodata.json): done
Solving environment: failed with current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - requests==2.14.2=py36_0

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

[shankari]

ok, so the obvious fix of upgrading to the more recent version works. But I am still not sure why this broke. The whole point of freezing dependencies is that you can wait to upgrade when you want to, and not deal with dependency hell.

Since this is not currently blocking anybody (correct, @kafitz?) I am going to file an issue against conda to understand what is going on. If using a conda environment with pinned dependencies has limitations, I want to understand what they are.

$ conda install requests==2.22.0=py36_0
Collecting package metadata (current_repodata.json): done
Solving environment: failed with current_repodata.json, will retry with next repodata source.
Initial quick solve with frozen env failed.  Unfreezing env and trying again.
Solving environment: done
Collecting package metadata (repodata.json): done
Solving environment: failed
Initial quick solve with frozen env failed.  Unfreezing env and trying again.
Solving environment: done

## Package Plan ##

  environment location: /home/shankari/miniconda3

  added / updated specs:
    - requests==2.22.0=py36_0

[kafitz]

It's possible to get it going, but on my current test server, I used either conda or pip without a pinned version to install requests. Since it's a relatively standalone package, it didn't require upping the version on anything else, but that's why it felt sketchy to me.

Luckily for the others, I was able to find the pinned version to manually download and install from the conda packages page via a Google search.

[shankari]

Filed conda/conda#9702 for conda

[shankari]

@kafitz I thought you said that you had the same behavior in the docker image, but that's not what I see. If I create a container based on the image from dockerhub, I can install requests in a new environment (conda/conda#9702 (comment))

[shankari]

it looks like this is related to the conda version (conda/conda#9702 (comment)). Using an older conda version (4.5.12) should work. Not sure how to specify the conda version in the environment to avoid this in the future.

[shankari]

I created a "dev" docker environment which embeds the correct conda version and works. Using that environment, or manually installing version 4.5.12 should work.

[lefterav]

I think the cleaner solution is what the conda developer suggest in the bug that you opened: to instruct users to reconfigure conda to allow the free sources
conda config --set restore_free_channel true

I did it and it seems to work.