[BUG] - Persisting conda environments via Docker volume

[rileyhun]

What docker image(s) are you using?

datascience-notebook

OS system and architecture running docker image

centos7

What Docker command are you running?

docker run -p 80:8888 \ -e CHOWN_HOME_OPTS='-R' \ -e CHOWN_HOME='yes' \ -v /tmp/opt/conda:/opt/conda \ daqml-jupyter:latest

How to Reproduce the problem?

1. Built docker file adapted from jupyter/datascience-notebook

ARG BASE_CONTAINER=jupyter/datascience-notebook:latest
FROM $BASE_CONTAINER

ARG NB_USER="jovyan"
ARG NB_UID="30000"
ARG NB_GID="1136"

ENV CONDA_DIR=/opt/conda \
    SHELL=/bin/bash \
    NB_USER=${NB_USER} \
    NB_UID=${NB_UID} \
    NB_GID=${NB_GID}

ENV PATH="${CONDA_DIR}/bin:${PATH}" \
    HOME="/home/${NB_USER}"

ENV JUPYTER_ENABLE_LAB=yes \
    JUPYTER_USER=${NB_USER} \
    JUPYTER_PASSWORD="" \
    JUPYTER_GRANT_SUDO=nopass

USER root

RUN echo "$JUPYTER_USER ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/$JUPYTER_USER

RUN fix-permissions /etc/jupyter/ "${HOME}" "${CONDA_DIR}"

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

# Install the widgets
RUN jupyter nbextension enable --py widgetsnbextension

VOLUME ${HOME}
VOLUME ${CONDA_DIR}

CMD ["/entrypoint.sh"]

entrypoint.sh

#!/bin/bash

echo "c.NotebookApp.token = ''" >> /home/jovyan/.jupyter/jupyter_notebook_config.py
echo "c.NotebookApp.port = 8888" >> /home/jovyan/.jupyter/jupyter_notebook_config.py
echo "c.NotebookApp.ip = '*'" >> /home/jovyan/.jupyter/jupyter_notebook_config.py
echo "c.NotebookApp.password = ''" >> /home/jovyan/.jupyter/jupyter_notebook_config.py

echo "c.ServerApp.token = ''" >> /home/jovyan/.jupyter/jupyter_server_config.py
echo "c.ServerApp.port = 8888" >> /home/jovyan/.jupyter/jupyter_server_config.py
echo "c.ServerApp.ip = '*'" >> /home/jovyan/.jupyter/jupyter_server_config.py
echo "c.ServerApp.password = ''" >> /home/jovyan/.jupyter/jupyter_server_config.py

start-notebook.sh

2. Build docker image docker build . -t daqml-jupyter -f Dockerfile
3. Run Container with volume mounted to local tmp directory

Command output

Entered start.sh with args: jupyter lab
Update jovyan's UID:GID to 30000:1136
userdel: group jovyan not removed because it is not the primary group of user jovyan.
Ensuring /home/jovyan is owned by 30000:1136 (chown options: -R)
Running as jovyan: jupyter lab
sudo: jupyter: command not found

Expected behavior

I expected the docker container to be running. Using a different volume, for example /home/jovyan seems to work fine - no issues there. It's just /opt/conda that seems unsuccessful.

Followed the instructions/tutorial here: https://ilperrozzi.medium.com/updating-packages-in-docker-containers-with-external-volumes-5923c23f664f

Actual behavior

The docker container does not start up. It fails and exits.

Anything else?

No response

[mathbunnyru]

I don't recommend mounting /opt/conda/ unless you really really need it.
If you don't need it, please, don't do this and everything should be fine.

Could you please tell me how you created your /tmp/opt/conda?
sudo: jupyter: command not found - does your local /tmp/opt/conda contain bin/ jupyter binary?
What permissions does it have?
There are so many things that can go wrong with this approach.

Run Container with volume mounted to local tmp directory
Please, provide the full command here.

[rileyhun]

Thank you @mathbunnyru for following up. Unfortunately, persisting conda environments is requirement from our users, unless there's another way for them to ensure their environments + kernels aren't wiped out every time we need to redeploy the infra on EC2 or do a reboot and restart the container

The full command is this:

docker run -p 80:8888 \ 
-e CHOWN_HOME_OPTS='-R' \ 
-e CHOWN_HOME='yes' \ 
-v /tmp/opt/conda:/opt/conda \ 
daqml-jupyter:latest

[mathbunnyru]

Please, answer my question about /tmp/opt/conda.
Also, could you please simplify your example and make it reproducible?
Get rid of apt packages / aws-cli / pip packages / etc, if they don't change anything.

[rileyhun]

/tmp folder already exists and is one of a handful of shared host directories docker can use by default. When you mount the docker volume, /opt/conda should be created in the tmp parent directory, but I just discovered that you need root access to create /opt/conda, so I might have just answered my question

[mathbunnyru]

This is why I was asking how /tmp/opt/conda was created ;)

[rileyhun]

It's still not working even if you use a different local path

Example:

docker run -p 80:8888 \                                
-e CHOWN_HOME_OPTS='-R' \
-e CHOWN_HOME='yes' \
-v /tmp/my_conda:/opt/conda \
daqml-jupyter:latest

But something like this, does work:

docker run -d -p 80:8888 \                                 
    -e CHOWN_HOME_OPTS='-R' \
    -e CHOWN_HOME='yes' \
    -v /tmp/user:/home/jovyan \
    daqml-jupyter:latest

Same error:

sudo: jupyter: command not found

Why is it looking for the jupyter command from within /opt/conda inside the docker container?

[mathbunnyru]

Once again, could you please answer these questions, I don't get it:

1. How you initially created your host /tmp/opt/conda directory?
2. Could you please share the contests of this folder? Both on a host machine and as a mounted volume.
   To see the content as a hosted volume, add ls /opt/conda in the end of you docker run.

[mathbunnyru]

I think your host directory was created incorrectly:
I did the following:

rm -rf /tmp/my_conda
id=$(docker create daqml-jupyter)
docker cp $id:/opt/conda /tmp/my_conda
rm -v $id

and then I was able to run jupyter (mounting /tmp/my_conda)

[rileyhun]

Thank you @mathbunnyru ,

Yes you're correct. I think the host directory was created incorrectly. Running your commands did the trick. Thank you for your help.