FileNotFoundError: [Errno 2] No such file or directory: 'marimo' when attempting to launch Marimo from single-user Docker container

[benz0li]

I tried with b-data's/my reference deployment at https://jupyter.b-data.ch, i.e.

pip install marimo
pip install git+https://github.com/jyio/jupyter-marimo-proxy.git

After restarting the server (container) and clicking on Marimo, I see the following in the logs:

[E 2024-06-21 13:29:45.219 ServerApp] Uncaught exception GET /user/benz0li/marimo/marimo/ ([redacted])
    HTTPServerRequest(protocol='http', host='jupyter.b-data.ch', method='GET', uri='/user/benz0li/marimo/marimo/', version='HTTP/1.1', remote_ip='[redacted]')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.12/site-packages/tornado/web.py", line 1790, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/jupyter_server_proxy/websocket.py", line 101, in get
        return await self.http_get(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/jupyter_server_proxy/handlers.py", line 725, in http_get
        return await ensure_async(self.proxy(self.port, path))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/jupyter_core/utils/__init__.py", line 198, in ensure_async
        result = await obj
                 ^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/jupyter_server_proxy/handlers.py", line 897, in proxy
        await self.ensure_process()
      File "/usr/local/lib/python3.12/site-packages/jupyter_server_proxy/handlers.py", line 884, in ensure_process
        await proc.start()
      File "/usr/local/lib/python3.12/site-packages/simpervisor/process.py", line 228, in start
        await self.proc.start()
      File "/usr/local/lib/python3.12/site-packages/simpervisor/process.py", line 81, in start
        self._proc = await asyncio.create_subprocess_exec(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/asyncio/subprocess.py", line 224, in create_subprocess_exec
        transport, protocol = await loop.subprocess_exec(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1744, in subprocess_exec
        transport = await self._make_subprocess_transport(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/asyncio/unix_events.py", line 211, in _make_subprocess_transport
        transp = _UnixSubprocessTransport(self, protocol, args, shell,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/asyncio/base_subprocess.py", line 36, in __init__
        self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
      File "/usr/local/lib/python3.12/asyncio/unix_events.py", line 820, in _start
        self._proc = subprocess.Popen(
                     ^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/subprocess.py", line 1026, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/usr/local/lib/python3.12/subprocess.py", line 1955, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'marimo'
[E 2024-06-21 13:29:45.313 ServerApp] {
      "X-Real-Ip": "[redacted]",
      "X-Forwarded-Server": "e77a48d46784",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "jupyter.b-data.ch",
      "X-Forwarded-For": "[redacted],::ffff:10.0.20.66",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Referer": "https://jupyter.b-data.ch/user/benz0li/marimo/lab",
      "Pragma": "no-cache",
      "Cookie": "_xsrf=[secret]; jupyterhub-user-benz0li-marimo=[secret]; jupyterhub-user-benz0li=[secret]; jupyterhub-session-id=[secret]; rl_anonymous_id=[secret]; rl_group_id=[secret]; rl_group_trait=[secret]; rl_trait=[secret]; rl_user_id=[secret]",
      "Cache-Control": "no-cache",
      "Accept-Language": "en-GB,en;q=0.9",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
      "Host": "jupyter.b-data.ch",
      "Connection": "keep-alive"
    }
[E 2024-06-21 13:29:45.313 ServerApp] 500 GET /user/benz0li/marimo/marimo/ (benz0li@[redacted]) 99.10ms

Cross reference:

• Marimo behind Jupyter / JupyterHub  marimo-team/marimo#905

Ping @jyio

[benz0li]

@jyio You may test yourself with b-data's/my demo deployment at https://demo.jupyter.b-data.ch.
ℹ️ Login with your GitHub account.

[jyio]

Hey, thank you for trying this out.

It looks like it has trouble finding marimo, which gets installed into ~/.local/bin/. The tricky bit is the $PATH that JupyterHub sees might be different from the $PATH that the shell sees. Both .zshrc and .bashrc prepend ~/.local/bin/ to $PATH, but I suspect JupyterHub doesn't.

For example, I have this line in my user containers' Dockerfile to allow the single-user JupyterHub to see what the users install:

ENV PATH=~/.local/bin:$PATH

Later, I could try making a variant that checks ~/.local/bin/ first.

[benz0li]

After setting

PATH=/home/benz0li/.local/bin:/opt/TinyTeX/bin/linux:/opt/quarto/bin:/opt/code-server/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`

and

pip install click

and clicking on Marimo, a new page opens showing

{"detail":"Not Found"}

but nothing more in the [container] logs.

[benz0li]

I think this needs more work.

---

With my JupyterLab Julia docker stack, one can simply execute

julia -e 'Pkg.add("Pluto")'
pip install jupyter-pluto-proxy

restart the server (container) and click on Pluto Notebook.

Everything works out of box – thanks to @yuvipanda.

---

Addendum:

The following startup files are in place for Julia: https://github.com/b-data/jupyterlab-julia-docker-stack/tree/97fa5731a5272d13261e7f1ece7ee737dd293a31/base/conf/julia/etc/skel/.julia/config
ℹ️ For more information, see https://github.com/b-data/jupyterlab-julia-docker-stack/blob/97fa5731a5272d13261e7f1ece7ee737dd293a31/NOTES.md#julia-startup-scripts

[jyio]

As suspected, PATH=/opt/TinyTeX/bin/linux:/opt/quarto/bin:/opt/code-server/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Here's a hack that finds Marimo in the search path prepended with ~/.local/bin/.

Try this:

$ pip install marimo click
$ pip install git+https://github.com/jyio/jupyter-marimo-proxy@user-local-bin

After restarting the container and clicking on Marimo, it works for me... but does it work for you? 🤔

[benz0li]

After restarting the container and clicking on Marimo, it works for me... but does it work for you? 🤔

No. Clicking on Marimo, a new page opens showing

{"detail":"Not Found"}

but nothing more in the [container] logs.

[benz0li]

IMHO PATH needs to be updated properly – and not just to find the exe for the command.

Like it is done in /usr/local/etc/ipython/ipython_config.py for my JupyterLab docker stacks: https://github.com/b-data/jupyterlab-python-docker-stack/blob/31e08865205b001330207cd1a17e5277c48e6a2c/base/conf/ipython/usr/local/etc/ipython/ipython_config.py#L3-L24
ℹ️ This ensures ~/.local/bin and ~/bin are not only taken into account by JupyterLab Terminals but also JupyterLab Notebooks and Consoles.

But even when providing a properly updated PATH, the page shows

{"detail":"Not Found"}

---

P.S.: It does not work with a non-localhost deployment of code-server using the marimo extension, either:

• code-server: Not allowed to display insecure content from http://localhost:2718/ marimo-team/vscode-marimo#27

[jyio]

But even when providing a properly updated PATH, the page shows

{"detail":"Not Found"}

Interesting. Your first post mentions /user/benz0li/marimo/marimo/ and /user/benz0li/marimo/lab in your production setup. So it seems my assumption that --base-url should be '/user/' + os.environ['JUPYTERHUB_USER'] + '/marimo' is incorrect. It would probably be better to make it os.environ['JUPYTERHUB_SERVICE_PREFIX'] + 'marimo' instead, like so.

Would you please try git+https://github.com/jyio/jupyter-marimo-proxy@service-prefix?

Edit: And, I agree that the $PATH issue is out of scope. This project should not make any assumptions about the user's (or admin's) search path priorities. But I'd probably leave that branch there for convenience's sake.

[benz0li]

Your first post mentions /user/benz0li/marimo/marimo/ and /user/benz0li/marimo/lab in your production setup.

/user/benz0li/marimo/ because I called my JuypterHub Named Server marimo.

/user/benz0li/marimo/marimo/ = The resulting URL when clicking on Marimo.

/user/benz0li/marimo/lab = The default URL of my setup, i.e. JupyterLab.

[benz0li]

No time for further testing. You can use https://demo.jupyter.b-data.ch as test environment.

[jyio]

Alright, so I set up a minimal JupyterHub with named servers, and JUPYTERHUB_SERVICE_PREFIX appears to work. Just let me know later if it works for you also 😉

[s-celles]

I did

$ pip install marimo click
$ pip install git+https://github.com/jyio/jupyter-marimo-proxy@user-local-bin

in a terminal at https://demo.jupyter.b-data.ch/ with Python (jupyterlab/python/scipy:latest) image

File / Hub Control Panel > Stop My Server then Start My Server (still with jupyterlab/python/scipy:latest) image)

I can see now Marimo icon

and run a simple sample Marimo notebook here also

[jyio]

Thank you @scls19fr for trying the search path hack.

#1 (comment)

IMHO PATH needs to be updated properly – and not just to find the exe for the command.

Like it is done in /usr/local/etc/ipython/ipython_config.py for my JupyterLab docker stacks: https://github.com/b-data/jupyterlab-python-docker-stack/blob/31e08865205b001330207cd1a17e5277c48e6a2c/base/conf/ipython/usr/local/etc/ipython/ipython_config.py#L3-L24 ℹ️ This ensures ~/.local/bin and ~/bin are not only taken into account by JupyterLab Terminals but also JupyterLab Notebooks and Consoles.

Here's how one might patch PATH directly, and it should be transmitted to children this way:

jupyter-marimo-proxy/jupyter_marimo_proxy/__init__.py

Line 8 in bc51eaa

'environment': { 'PATH': os.path.expanduser(os.path.join('~', '.local', 'bin')) + os.pathsep + os.environ.get('PATH', os.defpath) },

But, if you ask me about properly updating PATH, I'd have to say the right way is to do it as early in the chain of events as possible. Do it once in your Dockerfile, and everything inherits your intended PATH (with some exceptions). No need to futz around with PATH in .bashrc, .zshrc, your Julia config, your IPython config, your Marimo config, etc...

[benz0li]

But, if you ask me about properly updating PATH, I'd have to say the right way is to do it as early in the chain of events as possible. Do it once in your Dockerfile, and everything inherits your intended PATH (with some exceptions). No need to futz around with PATH in .bashrc, .zshrc, your Julia config, your IPython config, your Marimo config, etc...

Every path outside a user’s home directory: Yes.
Every path within a user’s home directory: No.

For example, I have this line in my user containers' Dockerfile to allow the single-user JupyterHub to see what the users install:

ENV PATH=~/.local/bin:$PATH

Prepending ~/.local/bin to PATH in the Dockerfile (i.e. hard-coding) is bad practice.
ℹ️ https://jupyterhub.readthedocs.io/en/stable/explanation/websecurity.html

---

PATH is not static. Conda would not work w/o manipulating PATH.

But even then, it’s complicated:

• Fix conda hook to work in both terminal and Jupyter Notebook jupyter/docker-stacks#2047
  • Fixes Conda environment not fully set in Jupyter jupyter/docker-stacks#2043
  • Fixes New conda environment issue with another username jupyter/docker-stacks#1792
• Use custom conda environment in Jupyter Notebook when user wants to use it jupyter/docker-stacks#2050

Really complicated:

• Adding support for activating conda environments in the kernel.json launch script? ipython/ipykernel#416
  • b-data's/my solution: b-data/jupyterlab-python-docker-stack@f9a9a50

[jyio]

For sure, it's complicated. There are lots of smart folks with strong opinions about what the search path should be, and I'm not one of them 😅

So let's just sort out the issue with the named servers because there's a straightforward solution (let me know if it works for you so we could lay this to rest).

And we could keep the special case with the ~/.local/bin search path prefix in its own branch, though I don't think it'd be a general solution for everyone (smart people, strong opinions, etc).

[jyio]

Here you go: https://github.com/jyio/jupyter-marimo-proxy/tree/feature-path-prefix-local-bin

[benz0li]

With b-data's/my JupyterLab docker stacks (or the demo deployment at https://demo.jupyter.b-data.ch), I recommend

pip install marimo click
pip install git+https://github.com/b-data/jupyter-marimo-proxy.git@jupyterlab-docker-stack

for now. Then restart the server (container) and click on Marimo.