Setup for Mac users - was conda: ResolvePackageNotFound

[lexnederbragt]

I am trying to install snakemake as per the instructions in https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/setup.html. When I run conda env update --file conda_env_all.yaml using https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/files/conda_env_all.yaml I get a ResolvePackageNotFound. error with a long list of packages:

Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:
  - libedit==3.1.20191231=he28a2e2_2
  - openjdk==10.0.2=h14c3975_1015
  - _openmp_mutex==4.5=1_gnu
  - zlib==1.2.11=h516909a_1010
  - pynacl==1.4.0=py39h3811e60_2
  - libsodium==1.0.18=h36c2ea0_1
  - tornado==6.1=py39h3811e60_1
  - bzip2==1.0.8=h7f98852_4
  - curl==7.76.1=h979ede3_1
  - zstd==1.4.9=ha95c52a_0
  - lcms2==2.11=hcbb858e_1
  - libopenblas==0.3.15=pthreads_h8fe5266_0
  - coincbc==2.10.5=hcee13e7_1
  - tk==8.6.10=h21135ba_1
  - libprotobuf==3.15.8=h780b84a_0
  - rich==10.1.0=py39hf3d152e_0
  - yarl==1.5.1=py39h07f9747_0
  - kallisto==0.46.2=h60f4f9f_2
  - libssh2==1.9.0=ha56f1ee_6
  - pthread-stubs==0.4=h36c2ea0_1001
  - googleapis-common-protos==1.53.0=py39hf3d152e_0
  - pango==1.42.4=h69149e4_5
  - freetype==2.10.4=h0708190_1
  - libstdcxx-ng==9.3.0=h6de172a_19
  - xorg-libxau==1.0.9=h7f98852_0
  - pillow==7.2.0=py39h6f3857e_2
  - wrapt==1.12.1=py39h3811e60_3
  - pygraphviz==1.7=py39h78163bd_0
  - markupsafe==1.1.1=py39h3811e60_3
  - xorg-libxext==1.3.4=h7f98852_1
  - lz4-c==1.9.3=h9c3ff4c_0
  - icu==68.1=h58526e2_0
  - fastx_toolkit==0.0.14=0
  - tbb==2020.2=h4bd325d_4
  - aiohttp==3.7.4=py39h3811e60_0
  - xorg-libxpm==3.5.13=h7f98852_0
  - aioeasywebdav==2.4.0=py39hf3d152e_1001
  - protobuf==3.15.8=py39he80948d_0
  - pcre==8.44=he1b5a44_0
  - multidict==5.1.0=py39h3811e60_1
  - pysocks==1.7.1=py39hf3d152e_3
  - pytest==6.2.4=py39hf3d152e_0
  - libglib==2.68.1=h3e27bee_0
  - importlib-metadata==4.0.1=py39hf3d152e_0
  - libpng==1.6.37=h21135ba_2
  - pulp==2.4=py39hf3d152e_0
  - setuptools==49.6.0=py39hf3d152e_3
  - google-crc32c==1.1.2=py39hb81f231_0
  - readline==8.1=h46c0cb4_0
  - chardet==4.0.0=py39hf3d152e_1
  - xorg-libxdmcp==1.1.3=h7f98852_0
  - libgfortran-ng==9.3.0=hff62375_19
  - ncurses==6.2=h58526e2_4
  - giflib==5.2.1=h36c2ea0_2
  - c-ares==1.17.1=h7f98852_1
  - expat==2.3.0=h9c3ff4c_0
  - gettext==0.19.8.1=h0b5b191_1005
  - libxml2==2.9.10=h72842e0_4
  - libgtextutils==0.7=h1b792b2_7
  - pixman==0.40.0=h36c2ea0_0
  - certifi==2021.10.8=py39hf3d152e_0
  - libuuid==2.32.1=h7f98852_1000
  - datrie==0.8.2=py39h3811e60_2
  - libtool==2.4.6=h58526e2_1007
  - libcroco==0.6.13=h5080e8c_2
  - libgfortran5==9.3.0=hff62375_19
  - matplotlib-base==3.4.1=py39h2fa2bec_0
  - simplejson==3.17.2=py39h3811e60_2
  - graphviz==2.42.3=h0511662_0
  - docutils==0.17.1=py39hf3d152e_0
  - grpcio==1.37.1=py39hff7568b_0
  - pyyaml==5.4.1=py39h3811e60_0
  - libwebp==1.0.2=h56121f0_5
  - glib==2.68.1=h9c3ff4c_0
  - xorg-libxrender==0.9.10=h7f98852_1003
  - xorg-kbproto==1.0.7=h7f98852_1002
  - fftw==3.3.9=nompi_h74d3f13_101
  - xz==5.2.5=h516909a_1
  - numpy==1.20.2=py39hdbf815f_0
  - xorg-renderproto==0.11.1=h7f98852_1002
  - jpeg==9d=h36c2ea0_0
  - pkg-config==0.29.2=h36c2ea0_1008
  - glib-tools==2.68.1=h9c3ff4c_0
  - cryptography==3.4.7=py39hbca0aa6_0
  - libiconv==1.16=h516909a_0
  - humanfriendly==9.1=py39hf3d152e_0
  - bcrypt==3.2.0=py39h3811e60_1
  - libgomp==9.3.0=h2828fa1_19
  - libnghttp2==1.43.0=h812cca2_0
  - xorg-libsm==1.2.3=hd9c2040_1000
  - pyrsistent==0.17.3=py39h3811e60_2
  - xorg-xproto==7.0.31=h7f98852_1007
  - jemalloc==5.2.1=h9c3ff4c_5
  - libcrc32c==1.1.1=h9c3ff4c_2
  - yaml==0.2.5=h516909a_0
  - _libgcc_mutex==0.1=conda_forge
  - librsvg==2.46.4=h33a7fed_0
  - openssl==1.1.1k=h7f98852_0
  - jbig==2.1=h7f98852_2002
  - fribidi==1.0.10=h36c2ea0_0
  - graphite2==1.3.13=h58526e2_1001
  - cython==0.29.23=py39he80948d_0
  - xorg-libice==1.0.10=h7f98852_0
  - boost-cpp==1.74.0=hc6e9bd1_2
  - xorg-libxt==1.2.1=h7f98852_2
  - libffi==3.3=h58526e2_2
  - psutil==5.8.0=py39h3811e60_1
  - xorg-libx11==1.7.0=h7f98852_0
  - libcurl==7.76.1=hc4aaa36_1
  - gobject-introspection==1.68.0=py39hcb793ab_1
  - fontconfig==2.13.1=hba837de_1005
  - ca-certificates==2021.10.8=ha878542_0
  - libxcb==1.13=h7f98852_1003
  - jupyter_core==4.7.1=py39hf3d152e_0
  - pandas==1.2.4=py39hde0f152_0
  - future==0.18.2=py39hf3d152e_3
  - libgcc==7.2.0=h69d50b8_2
  - salmon==1.4.0=h84f40af_1
  - perl==5.26.2=h36c2ea0_1008
  - sqlite==3.35.5=h74cdb3f_0
  - brotlipy==0.7.0=py39h3811e60_1001
  - cffi==1.14.5=py39he32792d_0
  - hdf5==1.10.6=nompi_h6a2412b_1114
  - libgcc-ng==9.3.0=h2828fa1_19
  - cairo==1.16.0=h6cf1ce9_1008
  - libtiff==4.1.0=hc3755c2_3
  - kiwisolver==1.3.1=py39h1a9c180_1
  - ld_impl_linux-64==2.35.1=hea4e1c9_2
  - pluggy==0.13.1=py39hf3d152e_4
  - openjpeg==2.3.1=hf7af979_3
  - krb5==1.17.2=h926e7f8_0
  - libev==4.33=h516909a_1
  - harfbuzz==2.8.1=h83ec7ef_0
  - xorg-xextproto==7.3.0=h7f98852_1002
  - python==3.9.2=hffdb5ce_0_cpython
  - coloredlogs==15.0=py39hf3d152e_0

That is 136 out of 239 packages (!). What can be done to sove this?

Additionally, isn't it quite unusual to specify so many packages in detail? Is there a 'minimal' set of packages that are required?

[tbooth]

Hi. I'm looking into this now. As you noted there was some issue with the file.

I've now updated the links and the file so this file should work OK:
https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/files/conda_env.yaml

Regarding the number of packages, this is the minimal list.

• fastx_toolkit=0.0.14
• salmon=1.4.0
• kallisto=0.46.2
• fastqc=0.11.9
• multiqc=1.10.1
• snakemake=5.32.2
• gnuplot=4.6.0
• graphviz=2.42.3
• graphicsmagick

The big list of packages was created by installing these into an environment, then doing "conda env export" to freeze the whole environment as conda_env.yaml. I believe this is the most reliable option since the decisions made by Conda regarding dependent packages will change over time, and my experience is that eventually these changes will break something.

However, there is now a pared-down version of the YAML file just containing the above list, if you want to install that and let Conda fill in the dependencies:
https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/files/conda_env_min.yaml

[tbooth]

Thanks @lexnederbragt for reporting this issue.

I believe this is fixed, but let me know if there are still problems.

[lexnederbragt]

Thanks! However, when I use https://carpentries-incubator.github.io/snakemake-novice-bioinformatics/files/conda_env_min.yaml as is, I get a lot (a lot!) of warnings about conflicts (Mac OS Big Sur, conda 4.12.0). When I instead used this, I got it to work (at least until episode 4).

name: snakemake_dash
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - fastqc
  - fastx_toolkit
  - gnuplot
  - graphicsmagick
  - graphviz
  - kallisto
  - multiqc
  - salmon
  - snakemake

Versions I got in my environment are:

fastqc=0.11.9
fastx_toolkit=0.0.14
gnuplot=5.4.3
graphicsmagick=1.3.37
graphviz=3.0.0
kallisto=0.48.0
multiqc=1.12
salmon=1.8.0
snakemake=7.6.2

So quite a number of newer versions than in conda_env_min.yaml...

I am going through the lesson and will get back if I see anything that didn't work as expected.

[tbooth]

It does not surprise me that several things do not quite work on the Mac, and it's great that you are able to test and fix these. I'll hold off merging the fixes and pull requests just now, but when you are done I'll roll them into a big fix and then we'll be able to say our course is good for Mac users. (I'm not sure it will ever be good for Windows users!)

[tbooth]

Hi @lexnederbragt once you are happy you have a working Conda setup for Mac then if you can conda env export it and share it here I will add it as an alternative conda_env_macos.yaml file in the repo.