Include mamba in miniforge?

[SylvainCorlay]

Mamba is much faster and consumes less memory than conda especially with large channels like conda-forge. This would solve the memory issues that a lot of people have experienced with e.g. RTD.

cc @wolfv

[rgommers]

+1 sounds interesting. Looks like you read someone's mind here @SylvainCorlay, we had a discussion on this exact topic a few days ago:)

To start with, are you thinking a set of installers based on mamba next to the current set based on conda? Could be nice to be able to evaluate them side-by-side.

[scopatz]

I think you could also shove mamba into the base installer.

Also, what would be really nice is if you could just configure the condarc file to use mamba whenever conda is called

[SylvainCorlay]

To start with, are you thinking a set of installers based on mamba next to the current set based on conda? Could be nice to be able to evaluate them side-by-side.

Just having it installed would allow people to use it, as an opt-in.

[isuruf]

mamba has a lot of different behaviour with conda that I don't want to replace. I used to use mamba and it updated my python version which took me a while to recover from and after 3 times I gave up.

[SylvainCorlay]

mamba has a lot of different behaviour with conda that I don't want to replace. I used to use mamba and it updated my python version which took me a while to recover from and after 3 times I gave up.

Do you have a minimal reproducer?

[SylvainCorlay]

Also, currently, from a fresh python 3.7 environment, conda install libsolv pybind11 pulls python 3.8, and conda install libsolv pybind11 python=3.7 does not give any conflict.

[isuruf]

Also, currently, from a fresh python 3.7 environment, conda install libsolv pybind11 pulls python 3.8, and conda install libsolv pybind11 python=3.7 does not give any conflict.

I guess you have a reproducer. conda doesn't upgrade your python unless explicitly told to do so.

[SylvainCorlay]

I guess you have a reproducer. conda doesn't upgrade your python unless explicitly told to do so.

this is conda. mamba behaves as expected on this one (and is much more reliable in my experience).

[isuruf]

I opened mamba-org/mamba#81, mamba-org/mamba#82, mamba-org/mamba#83

[isuruf]

And also mamba-org/mamba#38

[SylvainCorlay]

I opened mamba-org/mamba#81, mamba-org/mamba#82, mamba-org/mamba#83

thanks, I have responded to the individual tickets (marked 83 as a bug).

[rgommers]

mamba has a lot of different behaviour with conda that I don't want to replace

This is interesting (and being resolved), but it seems orthogonal to including mamba? You don't have to use it if conda is also still included.

[hmaarrfk]

What hardware specs did you have trouble with?

[hmaarrfk]

how do you currently install mamba? conda install mamba didn't work for me.

[wolfv]

conda install mamba -c conda-forge should work, and it's tested + used at least on Linux

[SylvainCorlay]

I opened mamba-org/mamba#81, mamba-org/mamba#82, mamba-org/mamba#83

We fixed:

• Feature minimization not respected mamba-org/mamba#81
• Python version change mamba-org/mamba#82
• Installing in non root env messes up -n mamba-org/mamba#83

[dhirschfeld]

IMHO mamba is really coming along in terms of production-quality stability. I'm sure there are still a number of rough edges but the best way to handle that would be to get actionable feedback by making it (more) easily available by packaging it alongside conda in miniforge

[hmaarrfk]

@SylvainCorlay i've been playing aroudn with mamba + miniforge for the last few weeks.

I noticed that it is recently poluting my directoris with pkgs folders. Any clue as to why that is happening?

[wolfv]

@hmaarrfk that was a bug which is fixed. What version of mamba are you using?

[hmaarrfk]

I guess I was on version 0.3.4. I guess I'm a typical "user" of "mamba", that never updates until they need to :D.

I can confirm that updating to 0.3.8 seems to alleviate the immiate issue (at least with a quick)

[nerohmot]

@wolfv as soon as TheSnakePit/mamba#83 is solved, i guess mamba can be added to miniforge right ?

[wolfv]

It would probably be possible to fix issue #83, although I think it's cleaner to install mamba in root.
Conda also does not work properly when not installed in the root environment.
And installing mamba in another environment will pull in a copy of conda (as mamba depends on conda).

[isuruf]

I'm still not convinced that we should include mamba in miniforge. miniforge is supposed to be the bare minimum needed to get conda and a python. Installing mamba is easy as conda install mamba. If we are going to install mamba what other software should we include?

[isuruf]

Perhaps build a miniforge-mamba?

[wolfv]

how about microforge with micromamba?

[rgommers]

I agree with the original rationale for this issue: mamba is way faster with conda-forge, and I consider conda effectively unusable for some use cases that require a lot of packages. Given that there's no significant improvements in conda on the horizon to make it approach mamba's performance, I'm very interested in (a) having mamba available by default and (b) maturing mamba faster through more usage. And mamba packages are only 500-600 kb in size. So I'd much prefer including it in baseline Miniforge.

[rgommers]

If we are going to install mamba what other software should we include?

I think mamba is a special case (and I can't think of other equivalent special cases), for one because it replaces the most special package in Miniforge (conda), and for another because it should be in root, while we prefer to teach users to not fill root with other stuff.

[isuruf]

how about microforge with micromamba?

+1. I like the name micro because there'll be no python I suppose.

(a) having mamba available by default

replaces the most special package in Miniforge

Given that mamba is not a replacement, but an addition, people have to explicitly use mamba. For eg: they have to use mamba install numpy instead of conda install numpy. So, I don't know what's hard about doing conda install mamba.

for another because it should be in root, while we prefer to teach users to not fill root with other stuff.

Why should it be in root? conda can be in non-root environments.

[nerohmot]

@isuruf ... and what kind of systems are you working on then that you can't spare a hand full of MB's?!?
... a hand full of MB's that you will spend sooner or later anyway 😒

[dhirschfeld]

If package size is a deal-breaker for some applications then maybe a separate package could work - mambaforge (or similar). That way users could choose a leaner miniforge or a fatter mambaforge which also included mamba and it's deps by default.

[scopatz]

I think we absolutely should have a community distro that includes mamba or micromamba. If we don't want that in miniforge, having a mambaforge that is also built as part of this repo would be fine with me

[isuruf]

That's fine with me. I suggested it in #23 (comment), but looks like @rgommers and @nerohmot wants it included in miniforge.

[rgommers]

If we're doing Mambaforge that'd work for me too. I think it's quite a bit more work and less ideal though - letting users choose at download time rather than later on seems sub-optimal.

[scopatz]

I think there is enough consensus to have a separate installer that we would feel comfortable merging in a PR to that effect.

I do think that there are two user stories going on here.

1. CI/CD where we want small and fast.
2. Downstream users who just want it to work, and don't care about speed or size. (Which is partially why the Anaconda distro exists)

We have 1 covered by miniforge and mambaforge. We don't really have 2 covered at all. I would be open to seeing proposals for a larger, more bloated, "just works" distribution. But that seems separate from the original issue here.

[isuruf]

I think it's quite a bit more work and less ideal though - letting users choose at download time rather than later on seems sub-optimal.

I don't get this. Users can choose to install mamba later on.

[rgommers]

Users can choose to install mamba later on.

I think you're ignoring the downsides of letting users jump through a hoop like that and letting them deal with two package management tools. It's like, why have an installer that defaults to conda-forge, when users can simply download Miniconda and change a config file to then default to conda-forge.

[nerohmot]

What are the chances of adding switches to the installer scripts ... something like :

• ./forge-install.sh --small ➜ smallest possible install (read: without mamba)
• ./forge-install.sh --medium ➜ small yet convenient base install (read: with mamba and maybe in future other stuff?)
• ./forge-install.sh --large ➜ big install where 95% of the users don't even need the available conda and mamba 😊 (read: the conda-forge alternative for anaconda)

[isuruf]

If somebody volunteers, I'm okay.

[nerohmot]

This ./forge-install.sh could be a 'front-end' for the current 8 (and growing i hope) individual packages there are now !
It could figure out the architecture (Linux/MacOS/... Intel/ARM/PPC ... 32/64 bit)on it's own, the install size from the switches, and the desired python implementation (CPython/ PyPy/...) also from switches ... based on the result download and install the exact installer.
@isuruf : do you also have size constraints on the size of the install script itself ?

[isuruf]

That's a different issue #39

[nerohmot]

two flies one stone 😉

[SylvainCorlay]

If we are going to install mamba what other software should we include?

I think mamba is a special case (and I can't think of other equivalent special cases), for one because it replaces the most special package in Miniforge (conda), and for another because it should be in root, while we prefer to teach users to not fill root with other stuff.

I think this is correct, which is that it goes alongside conda, and that we don't want to encourage users to install stuff in the base env.

[SylvainCorlay]

@minrk ran some benchmarks with mamba in repo2docker in a couple of repositories, and if we don't require a separate installation of mamba with conda, that will be a non-negligeable improvement in the binder build times (remove the green portion):

[isuruf]

It's always going to be a tradeoff between speed and storage. For binder, speed is important. For hosting in Heroku (where we host the webservice), storage is important. There's a 600MB limit in Heroku. (We switched to a different way in Heroku for the webservice and the storage is not an issue there anymore, but I've got another webapp using the older way)

I'd prefer if people don't put their needs above others and work on a solution for everybody.

[SylvainCorlay]

I'd prefer if people don't put their needs above others and work on a solution for everybody

Really not what I was doing, but adding some material on the interest of the question.

It's always going to be a tradeoff between speed and storage.

Hopefully, a micromamba-style installer will help with both speed and size eventually.

[hmaarrfk]

There isn't too much automation in this repo.

Maybe Mamba can take on releasing minimamba3??

[hadim]

My two cents to the debate.

If disk size is an issue in certain cases I would expect speed to be really more important for the majority of the people (including me it's true but I don't think anybody here is trying to put their needs above others).

It's true you can always install miniforge and then conda install mamba but I think the main drawback of this is that you still rely on conda to solve conda install mamba and yes it is slow. On GA using setup-miniconda, the conda install mamba step takes 7min which corresponds to about 85% of my CI time. Shipping mamba within miniforge would probably reduce CI time for a lot of people (not only me here).

That being said those kinds of performance issues are also being looked at on the setup-miniconda side (see conda-incubator/setup-miniconda#106).

[hmaarrfk]

i'm becoming more and more convinced of mamba.

We still dont have mamba build, but maybe that shouldn't hold things up.

[hadim]

I am using it for a few months now. It has been a game changer for me.

[wolfv]

@hmaarrfk you can try boa, it has "conda mambabuild ..." and "boa build ..." where the boa native recipe format is slightly updated from the conda one. You can find working example recipes in the mamba-org/boa-forge repository if you're curious.

[xhochy]

I made a Draft PR that would built Mambaforge in addition to Miniforge on this repo: #70

[hadim]

Thanks for the effort @SylvainCorlay and @xhochy. I can confirm CI with setup-conda is much faster now. See conda-incubator/setup-miniconda#89 (comment) for a snippet.

[SylvainCorlay]

We now have builds of miniforge available that include mamba (mambaforge).

Closing. Thanks everyone!