Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New conda environment issue with another username #1792

Closed
masip85 opened this issue Sep 29, 2022 · 12 comments · Fixed by #2047
Closed

New conda environment issue with another username #1792

masip85 opened this issue Sep 29, 2022 · 12 comments · Fixed by #2047
Labels
type:Bug A problem with the definition of one of the docker images maintained here

Comments

@masip85
Copy link

masip85 commented Sep 29, 2022

What docker image(s) are you using?

scipy-notebook

OS system and architecture running docker image

aarch64

What Docker command are you running?

docker run -it --name myproject --rm
-p 8888:8888
--user root
-e NB_USER=vicente
-e NB_UID="$(id -u)"
-e NB_GID="$(id -g)"
-e CHOWN_HOME=yes
-e CHOWN_HOME_OPTS="-R"
-w "/home/${NB_USER}"
-v "${PWD}"/test:/home/vicente/work
jupyter/myproject-notebook

How to Reproduce the problem?

  1. Create Dockerfile with conda environment following doc:
    https://jupyter-docker-stacks.readthedocs.io/en/latest/using/recipes.html#add-a-custom-conda-environment-and-jupyter-kernel
    adding at the end the commented line:
    RUN echo "conda activate ${conda_env}" >> "${HOME}/.bashrc"

  2. Execute docker run with new NB_USER.

Command output

No response

Expected behavior

I want my user has the conda environments in the list of kernels from the beginning (as is documented, but for any other user)

Actual behavior

# create Python kernel and link it to jupyter
RUN "${CONDA_DIR}/envs/${conda_env}/bin/python" -m ipykernel install --user --name="${conda_env}" && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"

and

RUN echo "conda activate ${conda_env}" >> "${HOME}/.bashrc" ,

in its context is executed only for jovyan home. So I guess is only being executed when jovyan starts a session. When my notebook starts with my user, if I open the terminal I only get default conda environment at the jupyter kernelspec list.

the And I dont want to use jovyan.

Anything else?

I can't use install without --user for install kernel it system wide:
RUN "${CONDA_DIR}/envs/${conda_env}/bin/python" -m ipykernel install --name="${conda_env}" && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}"
[Errno 13] Permission denied: '/usr/local/share/jupyter'

And in that case, I would have to hardcode this in dockerfile,
RUN echo "conda activate ${conda_env}" >> "/home/vicente/.bashrc"

which is not very pleasant. If you could solve the first issue, I'd like to get a nice way for doing the second one.

If the solution is not easy,maybe the best way is to giveup to conda environments, and always overwrite default conda environment with custom conda yaml or whatever. But in any of these cases,(if I am right) everything should be documented.

@masip85 masip85 added the type:Bug A problem with the definition of one of the docker images maintained here label Sep 29, 2022
@Bidek56
Copy link
Contributor

Bidek56 commented Sep 30, 2022

Can you use pip? This command works fine, and I can install packages with no issues.
docker run -it --name myproject --rm -p 8888:8888 --user root -e NB_USER=vicente -e NB_UID="$(id -u)" -e NB_GID="$(id -g)" -e CHOWN_HOME=yes -e CHOWN_HOME_OPTS="-R" -w "/home/${NB_USER}" -v "${PWD}":/home/vicente/work jupyter/pyspark-notebook

@mathbunnyru mathbunnyru changed the title New conda environmnet issue [BUG] - New conda environment issue Oct 2, 2022
@mathbunnyru
Copy link
Member

@Bidek56 I guess the problem here is that .bashrc is not run when changing username.
In your run command bashrc is more-or-less default, so you don't think any issues.
In the issue, your .bashrc is used to activate non-default environment.

@masip85
Copy link
Author

masip85 commented Oct 3, 2022

@mathbunnyru , Yes, I'd do it, but first ipykernel install must be successful for all users, and it's only done for jovyan. How should I solve this?

@mathbunnyru
Copy link
Member

@masip85 I understand this, my message was addressed to @Bidek56.

I don't know a good solution for your issue, but I might have a hack that might work (I haven't tested it).
Could you maybe mount /home/vicente folder, instead of the work folder and then put .bashrc in this folder with the activate line.

It's a bit ugly, because you will have all the config files in your mounted folder.
Also, I highly recommend making a backup, before testing, just in case.

@masip85
Copy link
Author

masip85 commented Oct 3, 2022 via email

@mathbunnyru
Copy link
Member

I don't know the answer yet, sorry :(

@masip85
Copy link
Author

masip85 commented Oct 7, 2022

This is a solution, not a perfect one, but the best one by now:
Jupyter Forum

USER 0:0
RUN sudo "${CONDA_DIR}/envs/${conda_env}/bin/python" -m ipykernel install --name="${conda_env}" && \
	fix-permissions "${CONDA_DIR}" && \
	fix-permissions "/home/${NB_USER}"
USER 1000:100

Now we could find a cleaner way for this:
Could you maybe mount /home/vicente folder, instead of the work folder and then put .bashrc in this folder with the activate line.

I don't understand why .bashrc is not functional as the doc says.This should copy it to my user:

    # Move or symlink the jovyan home directory to the desired users home
    # directory if it doesn't already exist, and update the current working
    # directory to the new location if needed.
    if [[ "${NB_USER}" != "jovyan" ]]; then
        if [[ ! -e "/home/${NB_USER}" ]]; then
            _log "Attempting to copy /home/jovyan to /home/${NB_USER}..."
            mkdir "/home/${NB_USER}"
            if cp -a /home/jovyan/. "/home/${NB_USER}/"; then
                _log "Success!"
            else
                _log "Failed to copy data from /home/jovyan to /home/${NB_USER}!"
                _log "Attempting to symlink /home/jovyan to /home/${NB_USER}..."
                if ln -s /home/jovyan "/home/${NB_USER}"; then
                    _log "Success creating symlink!"
                else
                    _log "ERROR: Failed copy data from /home/jovyan to /home/${NB_USER} or to create symlink!"
                    exit 1
                fi
            fi
        fi

@njacobson-nci
Copy link

I am also having this issue in a gpu-jupyter install. The cuda environment doesn't transfer over cleanly to the new user.

iot-salzburg/gpu-jupyter#108 (comment)

@benz0li
Copy link
Contributor

benz0li commented Mar 28, 2023

@njacobson-nci You may be interested in the CUDA-enabled JupyterLab Python docker stack, then:

[...]

Prerequisites

The same as the JupyterLab Python docker stack plus

  • NVIDIA GPU
  • NVIDIA Linux driver
  • NVIDIA Container Toolkit

ℹ️ The host running the GPU accelerated images only requires the NVIDIA driver, the CUDA toolkit does not have to be installed.

[...]

Similar projects

What makes this project different:

  1. Multi-arch: linux/amd64, linux/arm64/v8
  2. Derived from nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04
    • including development libraries and headers
  3. TensortRT and TensorRT plugin libraries
    • including development libraries and headers
  4. IDE: code-server next to JupyterLab
  5. Just Python – no Conda / Mamba

See Notes for tweaks, settings, etc.

@njacobson-nci
Copy link

@benz0li
Thanks, I will try out your library

@benz0li
Copy link
Contributor

benz0li commented Mar 29, 2023

@njacobson-nci Follow-up to iot-salzburg/gpu-jupyter#108 (comment). Using one of my images and after pip install bitsandbytes torch:

$ python
Python 3.10.10 (main, Feb  8 2023, 13:06:32) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bitsandbytes

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
/home/benz0li/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/extras/CUPTI/lib64'), PosixPath('/usr/local/cuda/lib'), PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')}
  warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /home/benz0li/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118.so...
>>> 

@mathbunnyru mathbunnyru changed the title [BUG] - New conda environment issue [BUG] - New conda environment issue with another username Aug 23, 2023
@mathbunnyru mathbunnyru changed the title [BUG] - New conda environment issue with another username New conda environment issue with another username Sep 10, 2023
@mathbunnyru
Copy link
Member

@masip85 I merged the fix.
New images will be ready in approximately 1 hour.
The recipe has also been updated.

Sorry for making you wait a year.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:Bug A problem with the definition of one of the docker images maintained here
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants