Replace conda with mamba #7227

crusaderky · 2021-02-16T11:33:05Z

Shave 2 minutes off each CI workflow
Blocked by #7383

xhochy · 2021-02-16T19:41:21Z

.github/workflows/ci-additional.yml

@@ -18,7 +18,8 @@ jobs:
      - name: Setup Conda Environment
        uses: conda-incubator/setup-miniconda@v2
        with:
-          miniconda-version: "latest"
+          mamba-version: "*"


Probably useful to have a look at the master of the setup-miniconda actions and use miniforge-variant: Mambaforge. This gives you a base environment with mamba pre-installed and conda-forge as a channel.

They are only yet available on master, not on v2. If they also work here well, setup-miniconda could do a new release to have them available in a stable fashion.

(ignore; I was still using @v2)

it works now, thanks

Warning: Unexpected input(s) 'miniforge-variant', 'miniforge-version', 'use-mamba'

crusaderky · 2021-02-17T14:28:36Z

There is a mamba-specific bug on MacOS Python 3.6 / numpy 1.15:

    try:
>       import scipy.stats
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   from .stats import *
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   from scipy.spatial.distance import cdist

../../../miniconda3/envs/test-environment/lib/python3.6/site-packages/scipy/stats/stats.py:174: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   from .qhull import *
E   ImportError: dlopen(/Users/runner/miniconda3/envs/test-environment/lib/python3.6/site-packages/scipy/spatial/qhull.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib
E     Referenced from: /Users/runner/miniconda3/envs/test-environment/lib/python3.6/site-packages/scipy/spatial/qhull.cpython-36m-darwin.so
E     Reason: image not found

../../../miniconda3/envs/test-environment/lib/python3.6/site-packages/scipy/spatial/__init__.py:98: ImportError

I'm parking this PR for now and I'm going to retest it after dropping Python 3.6

mrocklin · 2021-02-17T15:21:33Z

cc @jakirkham who might find this interesting

…

On Wed, Feb 17, 2021 at 8:28 AM crusaderky ***@***.***> wrote: There is a mamba-specific bug on MacOS Python 3.6 / numpy 1.15: try: > import scipy.stats _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > from .stats import * _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > from scipy.spatial.distance import cdist ../../../miniconda3/envs/test-environment/lib/python3.6/site-packages/scipy/stats/stats.py:174: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > from .qhull import * E ImportError: dlopen(/Users/runner/miniconda3/envs/test-environment/lib/python3.6/site-packages/scipy/spatial/qhull.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib E Referenced from: /Users/runner/miniconda3/envs/test-environment/lib/python3.6/site-packages/scipy/spatial/qhull.cpython-36m-darwin.so E Reason: image not found ../../../miniconda3/envs/test-environment/lib/python3.6/site-packages/scipy/spatial/__init__.py:98: ImportError I'm parking this PR for now and I'm going to retest it after dropping Python 3.6 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7227 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTFBDTG3I7FBCQ4H6ITS7PHCNANCNFSM4XWJIYUQ> .

jakirkham · 2021-02-17T19:17:38Z

Would suggest not using @master as future changes could break this workflow. If we want to use this functionality, would suggest raising an issue on that repo asking for a new tag.

xhochy · 2021-02-17T19:35:57Z

Raised this in conda-incubator/setup-miniconda#106 (comment)

thomasjpfan · 2021-02-19T19:24:11Z

.github/workflows/ci-additional.yml

@@ -16,9 +16,11 @@ jobs:
        uses: actions/checkout@v2

      - name: Setup Conda Environment
-        uses: conda-incubator/setup-miniconda@v2
+        uses: conda-incubator/setup-miniconda@master


If you want this now, you can pin this to a commit:

Suggested change

uses: conda-incubator/setup-miniconda@master

uses: conda-incubator/setup-miniconda@77b16ed746da28724c61e1f1ad23395a4b695ef5

and then make a follow up PR when a new version gets tagged.

crusaderky · 2021-03-12T08:24:24Z

xref conda-incubator/setup-miniconda#106

crusaderky · 2021-03-12T15:54:39Z

mamba is broken on numpy 1.15 regardless of Python version; see above.

crusaderky · 2021-03-13T11:05:48Z

I've split out the bump to numpy 1.16 into a separate PR, which is now a dependency of this one: #7383

jakirkham · 2021-03-29T17:21:21Z

continuous_integration/scripts/install.sh

+    # FIXME https://github.com/mamba-org/mamba/issues/412
+    # mamba uninstall --force numpy pandas
    conda uninstall --force numpy pandas


xref: mamba-org/mamba#412

crusaderky · 2021-03-29T21:49:40Z

@jrbourbeau ready for merge. Weirdly, unlike in dask/distributed I don't need to specify the channels in the workflow.

wolfv · 2021-03-30T07:46:48Z

@crusaderky cool to see this! Any info on what is broken with mamba / numpy 1.15? is it mamba or broken metadata or numpy? If there is something we can fix in mamba, please let us know!

crusaderky · 2021-03-30T10:09:24Z

@wolfv https://github.com/dask/dask/runs/2101922250

>   import scipy.stats
>   from .stats import *
>   from scipy.spatial.distance import cdist
>   from .qhull import *
E   ImportError: dlopen(/Users/runner/miniconda3/envs/test-environment/lib/python3.7/site-packages/scipy/spatial/qhull.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib
E     Referenced from: /Users/runner/miniconda3/envs/test-environment/lib/python3.7/site-packages/scipy/spatial/qhull.cpython-37m-darwin.so
E     Reason: image not found

this happens on MacOSX only, and only on mamba. conda list output is linked above.

wolfv · 2021-03-30T10:22:31Z

Ok, looks like mixing defaults & conda-forge channels.
Any reason for keeping defaults in there?

Otherwise, at least using strict-channel priority should help (as recommended by conda-forge).

crusaderky · 2021-03-30T10:32:55Z

@wolfv the exact same environment file works fine with conda.
defaults are there as a fallback for the cases when there isn't a package in conda-forge

crusaderky added 2 commits February 16, 2021 11:32

Replace conda with mamba

237ef9b

Merge remote-tracking branch 'upstream/master' into mamba

8b62892

xhochy reviewed Feb 16, 2021

View reviewed changes

crusaderky added 7 commits February 17, 2021 09:58

Use mambaforge

e0a8f22

Merge remote-tracking branch 'upstream/master' into mamba

0584356

Use mamba everywhere

c2f0203

fix Mambaforge

b1ff512

Do not use 'miniforge-variant', 'miniforge-version', 'use-mamba'

7ce4582

Warning: Unexpected input(s) 'miniforge-variant', 'miniforge-version', 'use-mamba'

Use master branch

59d1ac9

Hack around mamba#412

0c26048

Merge branch 'master' into mamba

82f413a

thomasjpfan reviewed Feb 19, 2021

View reviewed changes

bmerry mentioned this pull request Feb 27, 2021

triu and tril have wrong dtypes for some chunks in old version of numpy #7287

Closed

Base automatically changed from master to main March 8, 2021 20:19

crusaderky added 5 commits March 11, 2021 21:33

Merge branch 'main' into mamba

ea9953e

pin commit

7e84699

fix

a3d62c7

fix

c7225c1

Merge branch 'main' into mamba

bc3dde2

crusaderky force-pushed the mamba branch from c9e4499 to 89c34e0 Compare March 12, 2021 08:19

crusaderky changed the title ~~WIP replace conda with mamba~~ Replace conda with mamba Mar 12, 2021

crusaderky changed the title ~~Replace conda with mamba~~ Replace conda with mamba; bump numpy to >=1.16 Mar 12, 2021

crusaderky marked this pull request as ready for review March 12, 2021 15:54

xref setup-miniconda issue

6534fe9

Merge branch 'main' into mamba

184c938

crusaderky mentioned this pull request Mar 13, 2021

Require numpy >=1.16 #7383

Merged

crusaderky force-pushed the mamba branch from 6bb3459 to 184c938 Compare March 13, 2021 11:05

crusaderky changed the title ~~Replace conda with mamba; bump numpy to >=1.16~~ Replace conda with mamba Mar 13, 2021

crusaderky mentioned this pull request Mar 13, 2021

Replace conda with mamba dask/distributed#4585

Merged

crusaderky and others added 7 commits March 24, 2021 10:15

Merge remote-tracking branch 'upstream/main' into mamba

8260323

Force channels in workflow

3b59156

Merge branch 'main' of https://github.com/dask/dask into mamba

e728293

setup-miniconda v2.1.0

3687cbb

Merge remote-tracking branch 'upstream/main' into mamba

6e1a22f

Merge remote-tracking branch 'origin/mamba' into mamba

0834769

No need for explicit channels anymore

398e798

jakirkham reviewed Mar 29, 2021

View reviewed changes

crusaderky force-pushed the mamba branch from 8d5162c to 398e798 Compare March 29, 2021 17:23

jrbourbeau approved these changes Mar 29, 2021

View reviewed changes

jrbourbeau merged commit cb34b29 into dask:main Mar 29, 2021

crusaderky deleted the mamba branch March 29, 2021 21:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace conda with mamba #7227

Replace conda with mamba #7227

crusaderky commented Feb 16, 2021 •

edited

Loading

xhochy Feb 16, 2021

xhochy Feb 17, 2021

crusaderky Feb 17, 2021 •

edited

Loading

crusaderky Feb 17, 2021

crusaderky commented Feb 17, 2021

mrocklin commented Feb 17, 2021 via email

jakirkham commented Feb 17, 2021

xhochy commented Feb 17, 2021

thomasjpfan Feb 19, 2021

crusaderky commented Mar 12, 2021

crusaderky commented Mar 12, 2021

crusaderky commented Mar 13, 2021

jakirkham Mar 29, 2021

crusaderky commented Mar 29, 2021

wolfv commented Mar 30, 2021

crusaderky commented Mar 30, 2021 •

edited

Loading

wolfv commented Mar 30, 2021

crusaderky commented Mar 30, 2021

	uses: conda-incubator/setup-miniconda@master
	uses: conda-incubator/setup-miniconda@77b16ed746da28724c61e1f1ad23395a4b695ef5

Replace conda with mamba #7227

Replace conda with mamba #7227

Conversation

crusaderky commented Feb 16, 2021 • edited Loading

xhochy Feb 16, 2021

Choose a reason for hiding this comment

xhochy Feb 17, 2021

Choose a reason for hiding this comment

crusaderky Feb 17, 2021 • edited Loading

Choose a reason for hiding this comment

crusaderky Feb 17, 2021

Choose a reason for hiding this comment

crusaderky commented Feb 17, 2021

mrocklin commented Feb 17, 2021 via email

jakirkham commented Feb 17, 2021

xhochy commented Feb 17, 2021

thomasjpfan Feb 19, 2021

Choose a reason for hiding this comment

crusaderky commented Mar 12, 2021

crusaderky commented Mar 12, 2021

crusaderky commented Mar 13, 2021

jakirkham Mar 29, 2021

Choose a reason for hiding this comment

crusaderky commented Mar 29, 2021

wolfv commented Mar 30, 2021

crusaderky commented Mar 30, 2021 • edited Loading

wolfv commented Mar 30, 2021

crusaderky commented Mar 30, 2021

crusaderky commented Feb 16, 2021 •

edited

Loading

crusaderky Feb 17, 2021 •

edited

Loading

crusaderky commented Mar 30, 2021 •

edited

Loading