Fix image build and bump base image version #197

ableuler · 2021-10-25T18:04:19Z

This bumps the notebooks base image from lab-3.0.16 to lab-3.2.1, so a good look from the @SwissDataScienceCenter/poc squad to check if that breaks something would be appreciated.

rokroskar · 2021-10-25T19:22:23Z

docker/py/requirements.txt

@@ -6,5 +6,4 @@ jupyterlab~=3.0.0
 papermill~=2.3.0
 powerline-shell~=0.7.0
 requests>=2.20.0
-setuptools==57.5.0


why remove this? it will be globally available and would make sure that the user's environment uses the same setuptools as the renku venv.

you could also pin the pip and wheel versions in fact

Because I had pinned it only very recently when replacing pipx with a handmade virtual env, believing that this would determine also the version used inside the virtual env and thus prevent the 2-to-3 bug when installing renku.
This assumption was wrong, therefore I'm removing this line again.

ah I see, it always uses the latest?

We might want to go even further and strictly pin everything in the users environment, but for now I just wanted to fix the build and not pin a package version based on a renku requirement in an env that doesn't install renku.

Not really sure which version it will use for the venv if nothing is specified, but you can determine it through the virtualenv command (which I didn't know before but I was pleased to find out).

ok - is setuptools 57.5.0 still needed for renku to install properly? In that case I would say we should pin it also in the global environment in case the user wants to pip install it to use renku as a library in python

yes it is still needed. good point, I added it to the user python env.

gavin-k-lee · 2021-10-26T10:06:20Z

I can help out with a review. I notice that the py-batch build fails - should this be looked into first?

ableuler · 2021-10-26T19:43:55Z

I can help out with a review. I notice that the py-batch build fails - should this be looked into first?

Yes, that would be very much appreciated. Thanks a lot! 🙏

gavin-k-lee · 2021-10-28T08:42:55Z

@TaoSunVoyage do we need to replicate pinning the Renku version in the batch Docker? I.e. like this:

renkulab-docker/docker/py/Dockerfile

Lines 65 to 71 in 771cc49

    
           ARG RENKU_VERSION 
        
           RUN mkdir -p $HOME/.renku/bin && \ 
        
               virtualenv --pip 21.3.1 --setuptools 57.5.0 --wheel 0.36.2  --no-periodic-update $HOME/.renku/venv && \ 
        
               source $HOME/.renku/venv/bin/activate && \ 
        
               pip install --no-cache renku==$RENKU_VERSION sentry-sdk && \ 
        
               deactivate && \ 
        
               ln -s $HOME/.renku/venv/bin/renku $HOME/.renku/bin/renku

instead of

renkulab-docker/docker/batch/Dockerfile

Lines 23 to 28 in 771cc49

    
           RUN python3 -m pip install --no-cache pipenv 
        
           RUN virtualenv /share/.renku && \ 
        
               . /share/.renku/bin/activate && \ 
        
               pip install --no-cache renku && \ 
        
               deactivate && \ 
        
               ln -s /share/.renku/bin/renku /share/bin

TaoSunVoyage · 2021-10-28T08:54:02Z

@TaoSunVoyage do we need to replicate pinning the Renku version in the batch Docker? I.e. like this:

renkulab-docker/docker/py/Dockerfile

Lines 65 to 71 in 771cc49

ARG RENKU_VERSION

RUN mkdir -p $HOME/.renku/bin && \

virtualenv --pip 21.3.1 --setuptools 57.5.0 --wheel 0.36.2 --no-periodic-update $HOME/.renku/venv && \

source $HOME/.renku/venv/bin/activate && \

pip install --no-cache renku==$RENKU_VERSION sentry-sdk && \

deactivate && \

ln -s $HOME/.renku/venv/bin/renku $HOME/.renku/bin/renku

instead of

renkulab-docker/docker/batch/Dockerfile

Lines 23 to 28 in 771cc49

RUN python3 -m pip install --no-cache pipenv

RUN virtualenv /share/.renku && \

. /share/.renku/bin/activate && \

pip install --no-cache renku && \

deactivate && \

ln -s /share/.renku/bin/renku /share/bin

Thanks @gavin-k-lee, it looks nice.

just wondering if it's possible to simply copy the folder of $HOME/.renku/ from the base to /share/.renku in the batch? keep the same venv between base and batch, and easy to maintain (maybe).

gavin-k-lee · 2021-11-03T15:31:07Z

@TaoSunVoyage do we need to replicate pinning the Renku version in the batch Docker? I.e. like this:

renkulab-docker/docker/py/Dockerfile

Lines 65 to 71 in 771cc49

ARG RENKU_VERSION

RUN mkdir -p $HOME/.renku/bin && \

virtualenv --pip 21.3.1 --setuptools 57.5.0 --wheel 0.36.2 --no-periodic-update $HOME/.renku/venv && \

source $HOME/.renku/venv/bin/activate && \

pip install --no-cache renku==$RENKU_VERSION sentry-sdk && \

deactivate && \

ln -s $HOME/.renku/venv/bin/renku $HOME/.renku/bin/renku

instead of

renkulab-docker/docker/batch/Dockerfile

Lines 23 to 28 in 771cc49

RUN python3 -m pip install --no-cache pipenv

RUN virtualenv /share/.renku && \

. /share/.renku/bin/activate && \

pip install --no-cache renku && \

deactivate && \

ln -s /share/.renku/bin/renku /share/bin

Thanks @gavin-k-lee, it looks nice.

just wondering if it's possible to simply copy the folder of $HOME/.renku/ from the base to /share/.renku in the batch? keep the same venv between base and batch, and easy to maintain (maybe).

I think that could work! Do you want to try implementing that fix?

ableuler · 2021-11-15T09:10:31Z

@TaoSunVoyage any chance you could give it a try with the fix that you and @gavin-k-lee have discussed. We should try and get this fix to the users asap.

TaoSunVoyage · 2021-11-15T12:05:45Z

The simple copy doesn't work as the python executable in the venv is a symbolic link to the conda python. So, I reinstalled it in batch image instead.

ableuler

LGTM, @TaoSunVoyage feel free to approve and merge.

* fix: pin setuptools, pip, wheel and renku in renku virtualenv * fix: solve certificate problem by upgrading base image * chore: pin setuptools also in user python env * fix: renku install * fix: typo in base arg Co-authored-by: Tao Sun <[email protected]>

* fix: pin setuptools, pip, wheel and renku in renku virtualenv * fix: solve certificate problem by upgrading base image * chore: pin setuptools also in user python env * fix: install renku with venv in batch image * fix: typo in base arg Co-authored-by: Tao Sun <[email protected]>

Andreas Bleuler added 2 commits October 25, 2021 20:00

fix: pin setuptools, pip, wheel and renku in renku virtualenv

371dfff

fix: solve certificate problem by upgrading base image

98ab30c

ableuler linked an issue Oct 25, 2021 that may be closed by this pull request

git client struggling with new letsencrypt root certificate #196

Closed

ableuler requested review from rokroskar, gavin-k-lee and TaoSunVoyage October 25, 2021 18:05

rokroskar reviewed Oct 25, 2021

View reviewed changes

chore: pin setuptools also in user python env

771cc49

ableuler mentioned this pull request Nov 11, 2021

git client struggling with new letsencrypt root certificate #196

Closed

ableuler mentioned this pull request Nov 15, 2021

Fix renku not found error #193

Merged

TaoSunVoyage added 2 commits November 15, 2021 12:09

Fix renku install

ce4eb44

Fix typo in base arg

7e61ec6

TaoSunVoyage requested a review from rokroskar November 15, 2021 12:06

ableuler commented Nov 15, 2021

View reviewed changes

TaoSunVoyage approved these changes Nov 15, 2021

View reviewed changes

TaoSunVoyage merged commit 81da834 into master Nov 15, 2021

TaoSunVoyage deleted the 000-fix-build branch November 15, 2021 22:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix image build and bump base image version #197

Fix image build and bump base image version #197

ableuler commented Oct 25, 2021 •

edited

Loading

rokroskar Oct 25, 2021

rokroskar Oct 25, 2021

ableuler Oct 25, 2021

rokroskar Oct 25, 2021

ableuler Oct 25, 2021

ableuler Oct 25, 2021 •

edited

Loading

rokroskar Oct 25, 2021

ableuler Oct 26, 2021

gavin-k-lee commented Oct 26, 2021

ableuler commented Oct 26, 2021

gavin-k-lee commented Oct 28, 2021 •

edited

Loading

TaoSunVoyage commented Oct 28, 2021

gavin-k-lee commented Nov 3, 2021

ableuler commented Nov 15, 2021

TaoSunVoyage commented Nov 15, 2021

ableuler left a comment

Fix image build and bump base image version #197

Fix image build and bump base image version #197

Conversation

ableuler commented Oct 25, 2021 • edited Loading

rokroskar Oct 25, 2021

Choose a reason for hiding this comment

rokroskar Oct 25, 2021

Choose a reason for hiding this comment

ableuler Oct 25, 2021

Choose a reason for hiding this comment

rokroskar Oct 25, 2021

Choose a reason for hiding this comment

ableuler Oct 25, 2021

Choose a reason for hiding this comment

ableuler Oct 25, 2021 • edited Loading

Choose a reason for hiding this comment

rokroskar Oct 25, 2021

Choose a reason for hiding this comment

ableuler Oct 26, 2021

Choose a reason for hiding this comment

gavin-k-lee commented Oct 26, 2021

ableuler commented Oct 26, 2021

gavin-k-lee commented Oct 28, 2021 • edited Loading

TaoSunVoyage commented Oct 28, 2021

gavin-k-lee commented Nov 3, 2021

ableuler commented Nov 15, 2021

TaoSunVoyage commented Nov 15, 2021

ableuler left a comment

Choose a reason for hiding this comment

ableuler commented Oct 25, 2021 •

edited

Loading

ableuler Oct 25, 2021 •

edited

Loading

gavin-k-lee commented Oct 28, 2021 •

edited

Loading