[DISC]: mounted storage running docker on Ubuntu no writing permissions

[pmreyes2]

operating system:

• Linux 4.4.0-141-generic
• #167-Ubuntu SMP Wed Dec 5 10:40:15 UTC 2018
• x86_64
• GNU/Linux
  Docker version:
• Docker version 18.09.1, build 4c52b90

the storage added does not have writing permissions.
A workaround is giving writing permissions to all to the target folder in the host computer.

[asreimer]

This is only a problem for linux. It doesn't appear to be a problem in macOS nor Windows.

@pmreyes2 that's definitely a workaround that works, but we don't want users to have to change permissions on their system directories just so that the container can use the mounted location (how would this even work in Windows?).

I thought this was happening because we are using the -v option to mount volumes and by default, this performs a bind mount, so a possible solution is to use docker volumes instead of mounting. In the docker python SDK, this is done by using the mount keyword instead of the volumes keyword in create_container. So we would need to modify this code in Resen.py. But how? According to the SDK, mounts needs to be a list of docker.types.Mount class instances, but I'm not sure how you specify the host location to mount into the container. After spending some time digging, I don't think that mounting volumes will save us here.

So I did some more digging and realized that we can't have been the only one's who had this problem, specifically, we can't be the only one's using the jupyter-stacks docker images that have had this problem.

And yup, there's built in support for this!.

The way you fix this is by passing the environment variables NB_UID and NB_GID to the container when you run a command. BUT, this only works if you run the start.sh script

@pmreyes2 can you try adding -e NB_UID=$(id -u) -e NB_GID=$(id -g) to your docker run command on the Ubuntu system you are testing on? Make sure you run the command start.sh. You might have to run it like this:

. /usr/local/bin/start.sh command_to_execute

[pmreyes2]

This is the previous behavior:

$ resen
    ___ ___ ___ ___ _  _ 
   | _ \ __/ __| __| \| |
   |   / _|\__ \ _|| .` |
   |_|_\___|___/___|_|\_|

Resen 2019.1.0rc1 -- Reproducible Software Environment

[resen] >>> create_bucket b0
[resen] >>> add_image b0 2019.1.0rc1
[resen] >>> add_port b0 8001 8001
[resen] >>> add_storage b0 /PATH/TO/LOCAL/STORAGE /home/jovyan/work/remote1 rw
[resen] >>> start_bucket b0
[resen] >>> start_jupyter b0 8001 8001 --lab
Jupyter lab can be accessed in a browser at: http://localhost:8001/?token=abcdef...

Then in the jupyter notebook I start a terminal:

(base) jovyan@XXXXXXXX:~/work$ ls -l
total 4
drwxr-xr-x 3 1111 1111 4096 May 24 05:01 remote1
(base) jovyan@XXXXXXXX:~/work$ touch remote1/some_file
touch: cannot touch 'remote1/some_file': Permission denied

where 1111 are both the local NB_IUD and NB_GID. Notice that the folder remote1 does not have writing permissions.

Now, following @asreimer findings:

• Stop the bucket to stop jupyter [resen] >>> stop_bucket b0
• Start the bucket again [resen] >>> start_bucket b0
• Exit the resen tool. [resen] >>> exit
• Get the name of the container with $ docker ps. The name is resen_b0. Now we issue the following command to change the NB_IUD and NB_GID environment variables from the docker to the local user and group ids. It has to be done as root:

$ docker exec -e NB_UID=$(id -u) -e NB_GID=$(id -g) --user root resen_b0 /usr/local/bin/start.sh
/usr/local/bin/start.sh: ignoring /usr/local/bin/start-notebook.d/*

Set username to: jovyan
usermod: no changes
Set jovyan UID to: 1234

Now, coming back to the resen tool and start jupytrer from the still running bucket container:

$ resen
[resen] >>> start_jupyter b0 8001 8001 --lab

Starting a terminal from jupyter:

(base) jovyan@XXXXXXXX:~/work$ ls -l
total 4
drwxr-xr-x 3 jovyan jovyan 4096 May 24 05:38 remote1
(base) jovyan@XXXXXXXX:~/work$ ls -l remote1/some_empty_file
-rw-r--r-- 1 jovyan jovyan 0 May 24 05:53 remote1/some_empty_file

Now the remote1 folder has owner jovyan and the folder is writable.
It could be standard to execute the /usr/local/bin/start.sh script right after starting the bucket.

[asreimer]

This isn't resolved yet, because the standard start_bucket command in resen doesn't use /usr/local/bin/start.sh. We need to do this.