make mpie-cmti standalone

[mbruns91]

This PR makes the mpie_cmti image definition "standalone", i.e. the layering starts with jupyter/base-notebook:ubuntu-22.0.

Before, the image was drived from pyiron, which was itself derived from md, which was derived from base, which was the only standalone image definition in docker-stacks.
Especially for this image, this makes managing the env easier, which is important as the shared conda environment on our cluster is based on this.

Broken down, I made some changes to Dockerfile, concatenating neccessary parts from the Dockerfiles of the previous "inheritance-chain". Similarly, I concatenated the environment.yml from all those image definitions

[mbruns91]

@niklassiemer you think this is okay?

I started the container locally, ran mamba list and compared the output with the output of running the same command on cmti; they are the same.

@jan-janssen : Just added you to make you aware of this change.

[niklassiemer]

LGTM! We should only see if this survives on the CI (probably larger?)

[mbruns91]

Actually I think this image is smaller; I'm about to leave my desk, so I can check tomorrow or so.

The container started from the image defined here is basically empty.
Maybe (in another PR) we could add some notebooks for typical tests we do to check our environment. This way, one can do those checks easily from a local container instance. I'm thinking about the typical imports, plugins, ...; stuff like that.

[niklassiemer]

We need to resync the dependencies. Some of them decreased, but probably I did not find all of them. E.g. it seems that bokeh and maxvolpy are not in the list any more? I will check again the next days :)

[mbruns91]

Ok, so upon concatenating the environment.yml from the image definitions of pyiron, base and md there occurred some merge conflict. Upon resolving this I seem to have removed some packages accidentially... I hope the last commit fixed that.

[mbruns91]

Environment-wise everything seems to be okay now. But now the image(s) are too large.
The mpie_cmti image is not actrually too large per se. But now, as it's "standalone", it doesn't share any layers with the other images anymore, so the device volume (the runners diks space) is filled faster.
This was already an issue in the past, right? I think we could circumvent this, if not all steps of the building/testing were performed on the same runner instance.

We could either

• try to only build & test the images derived from the files on the affected directory of a PR (e.g. in this case mpie_cmti only). I think this could be a bit ticky but doable. However it would increase the complexity of the CI/CD pipeline.
• make seperate repos for each image (and make "docker-stacks" an organization?).

What do you think, @niklassiemer?

[niklassiemer]

I would not make an org for such few repos, and I also do not even think we need multiple repos. For now, we can "just" use different CI runs, one for each scenario. The main thing would be how to build and push all docker images in case of a release... maybe we need to split the releases like "cmti_mpie_${date}" release instead of only "${date}" ones

[niklassiemer]

Now I tried to split the testing to two VMs like before in #347. Back then I failed (and the issue was not to pressing) since all the builds needed to be in one VM. Now, this is not intended this way any longer!

[mbruns91]

Let's merge this PR and do the splitting for the test and release in another PR. @niklassiemer , can you please approve the changes?

[niklassiemer]

Hmm... the code in test and push is quite similar and I would expect the push to break due to the changes in the cmti/Dockerfile. Thus, let me take over the responsibility and I try to get the release action aligned with the test action before merging it in.

[mbruns91]

Yeah, you're right. You really want to do it? I can take care of it as well.

[niklassiemer]

Ok, if you want to play a bit with the github CI, go ahead :) There are actions using caching and or artifacts which can be used to combine things from different jobs. Just ping me for any question 👍

[mbruns91]

I really don't like to do all of this in this PR, it appears messy to me.

I opened another (draft) PR where I'm working on the CI improvements and applied the same changes done here to testing.yml.

[niklassiemer]

The problem is a bit that these things are indeed coupled... i.e. you need the Dockerfile change to build standalone and you need the ci change that this fits into one VM... probably we need stacked PRs to have it clean :)