running in jupyterhub inside docker image

[scivm]

I can see this would be useful running from jupyterhub which is a multi tenant jupyter service. For example, a user runs a local notebook and sends a job to a remote cluster. I would like the databricks cluster to start when the job is sent and not be running continuously and then shutdown after the clusters inactive timeout settiing.

In our case we use jupyterhub running in AKS and Azure Databricks in own vnet.

I did try to create a docker image as an extension of one of the default jupyter images.

`

FROM jupyter/datascience-notebook:latest
ENV BASH_ENV ~/.bashrc
RUN conda create -n db-jlab
RUN echo "source activate db-jlab" > ~/.bashrc
ENV PATH /opt/conda/envs/env/bin:$PATH
RUN pip install --upgrade databrickslabs-jupyterlab
RUN databrickslabs-jupyterlab -b
RUN pip install databricks-cli
USER $NB_USER

`
I am still testing

[bernhard-42]

Hi @scivm, interesting approach, I haven't thought about jupyterhub up to now. Did you manage to get it running.
And, btw., for running jobs on Databricks that start a cluster and shut it down afterwards, there would also be the Jobs REST API. Have you tried this one?

[bernhard-42]

@scivm for your information, the upcoming V2 of Jupyterlab Integration will have docker support.
see https://github.com/databrickslabs/jupyterlab-integration/tree/master/docker/image and https://github.com/databrickslabs/jupyterlab-integration/blob/master/README_v2.md#2-running-with-docker
I need to do a few further tests and will then publish V2 (rc2 is already on pypi, dockerhub will follow as soon as V2 is released)

[bernhard-42]

docker is supported. If this is not sufficient, please reopen.