conda fails to install python=36 (verification error)

[ewenmcneill]

On an Ubuntu 16.04 system which had previously had litex-buildenv installed, updating to the latest commit (52f75717ac6) which switches to python-36 results in a system that:

• Fails scripts/enter-env.sh because python 3.6 is missing; and

• Fails scripts/download-env.sh because conda fails to install python 3.6 due to CondaVerificationError on seemingly every single python file, eg

  CondaVerificationError: The package for chardet located at /src/fpga-miniconf/litex-buildenv/build/conda/pkgs/chardet-3.0.4-py36h0f667ec_1
  appears to be corrupted. The path 'lib/python3.6/site-packages/chardet/pycache/init.cpython-36.pyc'
  specified in the package manifest cannot be found.

(see also gist with full output).

From some reading (eg, conda/conda-build#1301, conda/conda#6033, conda-forge/python-feedstock#70) my current guess is that perhaps this was an issue with an older version of conda, and then the bug got fixed, and everyone moved on. However it appears that scripts/download-env.sh will only ever update conda on first installation (from miniconda) and then after that assumes that the installed version will work perfectly; see

litex-buildenv/scripts/download-env.sh

Lines 192 to 208 in 52f7571

	echo "Installing conda (self contained Python environment with binary package support)"
	if [[ ! -e $CONDA_DIR/bin/conda ]]; then
	cd $BUILD_DIR
	# FIXME: Get the miniconda people to add a "self check" mode
	wget --no-verbose --continue https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
	chmod a+x Miniconda3-latest-Linux-x86_64.sh
	# -p to specify the install location
	# -b to enable batch mode (no prompts)
	# -f to not return an error if the location specified by -p already exists
	./Miniconda3-latest-Linux-x86_64.sh -p $CONDA_DIR -b -f
	fix_conda
	conda config --system --set always_yes yes
	conda config --system --set changeps1 no
	conda config --system --add envs_dirs $CONDA_DIR/envs
	conda config --system --add pkgs_dirs $CONDA_DIR/pkgs
	conda update -q conda
	fi

(and note the conda ... update is guarded by if no bin/conda exists, so it'll only happen once).

I'm about to try manually doing conda update conda and see if that is sufficient to fix the issue. But am making an issue now as a record in case others run into the same issue...

Ewen

[ewenmcneill]

conda update conda fails in basically the same way; many many reports of __pycache__/*.cpython-36.pyc, eg:

(LX P=arty C=or1k.linux F=linux R=in-tftpd-fix) ewen@parthenon:/src/fpga-minicon
f/litex-buildenv$ conda update conda
Solving environment: done

## Package Plan ##

  environment location: /src/fpga-miniconf/litex-buildenv/build/conda

  added / updated specs: 
    - conda


The following packages will be UPDATED:

    asn1crypto:       0.24.0-py35_0            --> 0.24.0-py36_0         
    certifi:          2017.11.5-py35h9749603_0 --> 2018.1.18-py36_0      
    cffi:             1.11.2-py35hc7b2db7_0    --> 1.11.4-py36h9745a5d_0 
    chardet:          3.0.4-py35hb6e9ddf_1     --> 3.0.4-py36h0f667ec_1  
    conda:            4.4.7-py35_0             --> 4.4.8-py36_0          
    cryptography:     2.1.4-py35hbeb2da1_0     --> 2.1.4-py36hd09be54_0  
    decorator:        4.1.2-py35h3a268aa_0     --> 4.2.1-py36_0          
    idna:             2.6-py35h8605a33_1       --> 2.6-py36h82fb2a8_1    
    ipython:          6.2.1-py35hd850d2a_1     --> 6.2.1-py36h88c514a_1  
    ipython_genutils: 0.2.0-py35hc9e07d0_0     --> 0.2.0-py36hb52b0d5_0  
    jedi:             0.11.0-py35_2            --> 0.11.1-py36_0         
    parso:            0.1.1-py35h1b200a3_0     --> 0.1.1-py36h35f843b_0  
    pexpect:          4.3.1-py35_0             --> 4.3.1-py36_0          
    pickleshare:      0.7.4-py35hd57304d_0     --> 0.7.4-py36h63277f8_0  
    pip:              9.0.1-py35h7e7da9d_4     --> 9.0.1-py36h6c6f9ce_4  
    prompt_toolkit:   1.0.15-py35hc09de7a_0    --> 1.0.15-py36h17d85b1_0 
    ptyprocess:       0.5.2-py35h38ce0a3_0     --> 0.5.2-py36h69acd42_0  
    pycosat:          0.6.3-py35h6b6bb97_0     --> 0.6.3-py36h0a5515d_0  
    pycparser:        2.18-py35h61b3040_1      --> 2.18-py36hf9f622e_1   
    pygments:         2.2.0-py35h0f41973_0     --> 2.2.0-py36h0d3125c_0  
    pyopenssl:        17.5.0-py35h4f8b8c8_0    --> 17.5.0-py36h20ba746_0 
    pyserial:         3.4-py35h84edd1e_0       --> 3.4-py36h08da44b_0    
    pysocks:          1.6.7-py35h6aefbb0_1     --> 1.6.7-py36hd97a5b1_1  
    python:           3.5.4-h417fded_24        --> 3.6.0-0               
    requests:         2.18.4-py35hb9e6ad1_1    --> 2.18.4-py36he2e5f8d_1 
    ruamel_yaml:      0.11.14-py35h8e2c16b_2   --> 0.15.35-py36h14c3975_1
    setuptools:       36.5.0-py35ha8c1747_0    --> 38.4.0-py36_0         
    simplegeneric:    0.8.1-py35h2ec4104_0     --> 0.8.1-py36h2cb9092_0  
    six:              1.11.0-py35h423b573_1    --> 1.11.0-py36h372c433_1 
    traitlets:        4.3.2-py35ha522a97_0     --> 4.3.2-py36h674d592_0  
    urllib3:          1.22-py35h2ab6e29_0      --> 1.22-py36hbe7ace6_0   
    wcwidth:          0.1.7-py35hcd08066_0     --> 0.1.7-py36hdf4376a_0  
    wheel:            0.30.0-py35hd3883cf_1    --> 0.30.0-py36hfd4bba0_1 

The following packages will be DOWNGRADED:

    readline:         7.0-ha6073c6_4           --> 6.2-2                 
    sqlite:           3.20.1-hb898158_2        --> 3.13.0-0              
    tk:               8.6.7-hc745277_3         --> 8.5.18-0              

Preparing transaction: done
Verifying transaction: failed

CondaVerificationError: The package for chardet located at /src/fpga-miniconf/litex-buildenv/build/conda/pkgs/chardet-3.0.4-py36h0f667ec_1
appears to be corrupted. The path 'lib/python3.6/site-packages/chardet/__pycache__/__init__.cpython-36.pyc'
specified in the package manifest cannot be found.
[...]

And digging further, these paths are indeed specified within the info/paths.json in the conda packages:

ewen@parthenon:/src/fpga-miniconf/litex-buildenv/build/conda$ find . -name "paths.json" | xargs grep "__pycache__/.*python-36.pyc" | head
./pkgs/chardet-3.0.4-py36h0f667ec_1/info/paths.json:      "_path": "lib/python3.6/site-packages/chardet/__pycache__/__init__.cpython-36.pyc",
./pkgs/chardet-3.0.4-py36h0f667ec_1/info/paths.json:      "_path": "lib/python3.6/site-packages/chardet/__pycache__/big5freq.cpython-36.pyc",
./pkgs/chardet-3.0.4-py36h0f667ec_1/info/paths.json:      "_path": "lib/python3.6/site-packages/chardet/__pycache__/big5prober.cpython-36.pyc",
./pkgs/chardet-3.0.4-py36h0f667ec_1/info/paths.json:      "_path": "lib/python3.6/site-packages/chardet/__pycache__/chardistribution.cpython-36.pyc",
./pkgs/chardet-3.0.4-py36h0f667ec_1/info/paths.json:      "_path": "lib/python3.6/site-packages/chardet/__pycache__/charsetgroupprober.cpython-36.pyc",
./pkgs/chardet-3.0.4-py36h0f667ec_1/info/paths.json:      "_path": "lib/python3.6/site-packages/chardet/__pycache__/charsetprober.cpython-36.pyc",
./pkgs/chardet-3.0.4-py36h0f667ec_1/info/paths.json:      "_path": "lib/python3.6/site-packages/chardet/__pycache__/codingstatemachine.cpython-36.pyc",
./pkgs/chardet-3.0.4-py36h0f667ec_1/info/paths.json:      "_path": "lib/python3.6/site-packages/chardet/__pycache__/compat.cpython-36.pyc",
./pkgs/chardet-3.0.4-py36h0f667ec_1/info/paths.json:      "_path": "lib/python3.6/site-packages/chardet/__pycache__/cp949prober.cpython-36.pyc",
./pkgs/chardet-3.0.4-py36h0f667ec_1/info/paths.json:      "_path": "lib/python3.6/site-packages/chardet/__pycache__/enums.cpython-36.pyc",
xargs: grep: terminated by signal 13
ewen@parthenon:/src/fpga-miniconf/litex-buildenv/build/conda$ 

although interestingly they're supposed to be a hardlink:

    {
      "_path": "lib/python3.6/site-packages/chardet/__pycache__/__init__.cpython-36.pyc",
      "path_type": "hardlink",
      "sha256": "2f491902f2616b7cf6d0dcfd1ef838edfe8533a27c8d825a02e9a77d744570a8",
      "size_in_bytes": 782
    },

I have no lib/python3.6:

ewen@parthenon:/src/fpga-miniconf/litex-buildenv/build/conda$ ls -d lib/python*
lib/python3.5
ewen@parthenon:/src/fpga-miniconf/litex-buildenv/build/conda$ 

(presumably because this install attempt fails).

Ewen

[ewenmcneill]

In case it helps, I have conda 4.4.7 installed, using Python 3.5, along with these packages:

(LX P=arty C=or1k.linux F=linux R=in-tftpd-fix) ewen@parthenon:/src/fpga-minicon
f/litex-buildenv$ conda --version
conda 4.4.7
(LX P=arty C=or1k.linux F=linux R=in-tftpd-fix) ewen@parthenon:/src/fpga-minicon
f/litex-buildenv$ conda list
# packages in environment at /src/fpga-miniconf/litex-buildenv/build/conda:
#
asn1crypto                0.24.0                   py35_0  
binutils-lm32-elf         2.28.0          20180115_065451    timvideos
binutils-or1k-elf         2.28.0          20180108_070246    timvideos
ca-certificates           2017.08.26           h1d4fec5_0  
certifi                   2017.11.5        py35h9749603_0  
cffi                      1.11.2           py35hc7b2db7_0  
chardet                   3.0.4            py35hb6e9ddf_1  
cloog                     0.18.0                        0  
colorama                  0.3.9                     
conda                     4.4.7                    py35_0  
conda-env                 2.6.0                h36134e3_1  
cryptography              2.1.4            py35hbeb2da1_0  
decorator                 4.1.2            py35h3a268aa_0  
flterm                    2.4_24_gee93960               0    timvideos
gcc-lm32-elf-newlib       5.4.0           20171225_070346    timvideos
gcc-lm32-elf-nostdc       5.4.0           20171211_070518    timvideos
gcc-or1k-elf-newlib       5.4.0_4334_g9310fdc97ee 20171211_071240    timvideos
gcc-or1k-elf-nostdc       5.4.0_4334_g9310fdc97ee 20171211_071023    timvideos
gmp                       6.1.2                h6c8ec71_1  
hdmi2usb.modeswitch       0.0.1                     
hexfile                   0.1.1                     
idna                      2.6              py35h8605a33_1  
ipython                   6.2.1            py35hd850d2a_1  
ipython_genutils          0.2.0            py35hc9e07d0_0  
isl                       0.12.2                        0  
jedi                      0.11.0                   py35_2  
libedit                   3.1                  heed3624_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 7.2.0                h7cc24e2_2  
libstdcxx-ng              7.2.0                h7a57d05_2  
litedram                  0.1                       
liteeth                   0.1                       
litepcie                  1.0                       
litesata                  1.0                       
litescope                 0.1                       
liteusb                   0.1                       
litevideo                 0.1                       
litex                     0.1                       
mpc                       1.0.3                hec55b23_5  
mpfr                      3.1.5                h11a74b3_2  
ncurses                   6.0                  h9df7e31_2  
openocd                   0.10.0_267_gf7836bbc7   20180115_0740    timvideos
openssl                   1.0.2n               hb7f436b_0  
parso                     0.1.1            py35h1b200a3_0  
pexpect                   4.3.1                    py35_0  
pickleshare               0.7.4            py35hd57304d_0  
pip                       9.0.1            py35h7e7da9d_4  
progressbar2              3.34.3                    
prompt_toolkit            1.0.15           py35hc09de7a_0  
ptyprocess                0.5.2            py35h38ce0a3_0  
pycosat                   0.6.3            py35h6b6bb97_0  
pycparser                 2.18             py35h61b3040_1  
pygments                  2.2.0            py35h0f41973_0  
pyopenssl                 17.5.0           py35h4f8b8c8_0  
pyserial                  3.4              py35h84edd1e_0  
pysocks                   1.6.7            py35h6aefbb0_1  
python                    3.5.4               h417fded_24  
python-utils              2.2.0                     
readline                  7.0                  ha6073c6_4  
requests                  2.18.4           py35hb9e6ad1_1  
ruamel_yaml               0.11.14          py35h8e2c16b_2  
setuptools                36.5.0           py35ha8c1747_0  
simplegeneric             0.8.1            py35h2ec4104_0  
six                       1.11.0           py35h423b573_1  
sqlite                    3.20.1               hb898158_2  
tk                        8.6.7                hc745277_3  
traitlets                 4.3.2            py35ha522a97_0  
urllib3                   1.22             py35h2ab6e29_0  
wcwidth                   0.1.7            py35hcd08066_0  
wheel                     0.30.0           py35hd3883cf_1  
xz                        5.2.3                h55aa19d_2  
yaml                      0.1.7                had09818_2  
zlib                      1.2.11               ha838bed_2  
(LX P=arty C=or1k.linux F=linux R=in-tftpd-fix) ewen@parthenon:/src/fpga-minicon
f/litex-buildenv$ 

all of which are notably the py35 versions rather than the py36 versions...

Ewen

[ewenmcneill]

If I remove conda-meta/pinned, which scripts/download-env.sh sets to python ==3.6, then I can successfully update conda to the latest version 4.4.8:

(LX P=arty C=or1k.linux F=linux R=in-tftpd-fix) ewen@parthenon:/src/fpga-minicon
f/litex-buildenv$ conda update conda
Solving environment: done

## Package Plan ##

  environment location: /src/fpga-miniconf/litex-buildenv/build/conda

  added / updated specs: 
    - conda


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-4.4.8                |           py35_0         937 KB
    certifi-2018.1.18          |           py35_0         143 KB
    ------------------------------------------------------------
                                           Total:         1.1 MB

The following packages will be UPDATED:

    certifi: 2017.11.5-py35h9749603_0 --> 2018.1.18-py35_0
    conda:   4.4.7-py35_0             --> 4.4.8-py35_0    


Downloading and Extracting Packages
conda 4.4.8: ########################################################### | 100% 
certifi 2018.1.18: ##################################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(LX P=arty C=or1k.linux F=linux R=in-tftpd-fix) ewen@parthenon:/src/fpga-minicon
f/litex-buildenv$ 

Unfortunately then conda install python=3.6 then fails in pretty much the same way as before, even without the pinned version, because it still wants to update all the py35 packages, and fails because the pyc files it expects aren't found :-(

Ewen

[ewenmcneill]

It appears others have had this sort of problem before, although not precisely like this:

• https://stackoverflow.com/questions/41535881/how-do-i-upgrade-to-python-3-6-with-conda

• https://www.scivision.co/switch-anaconda-python-36/

and the fix they chose was basically "throw away that environment and start again". It looks like in theory upgrading python is now supposed to work (eg, documented at https://conda.io/docs/user-guide/tasks/manage-python.html#updating-or-upgrading-python), but in practice it's not clear that it will actually manage a major version transition with dependent packages...

Ewen

[ewenmcneill]

After concluding that reinstalling the conda environment was going to be quicker, I did:

mv build/conda build/conda-3.5
# set environment variables
scripts/download-env.sh

and it seemed to work perfectly. I assume the python=3.5 to python=3.6 upgrade process isn't very well tested. I did briefly try to find a combination of conda packages I could remove to avoid the .pyc complaints... but a bunch that I tried were conda dependencies, and unsurprisingly conda is unwilling to remove itself, so I gave up on that approach.

FTR, output of reinstalling after mv build/conda build/conda-3.5. At least with a fast network connection it took only a few minutes -- way less time than I spent trying to make it work without doing that :-) (And it might have been faster if I had, eg, found a way to save conda/pkgs/ or similar.)

Ewen

[mithro]

I'm going to close this as we have moved to Python 3.6 now....