hooks.d running stat on volumes before hooks can do volume work

[ghost]

Based on reading this and this, I believe the following should run the mount-fixer script before performing a stat on anything.

Edit: I have made steps in this gist that use the default hooks.d path so that --hooks-dir does not need to be used.

Where this was ran

This is all being done inside the podman machine on a macbook.

$ podman machine init
$ podman machine start
$ podman machine ssh

Setup

Directories for hooks and executables.

mkdir -p  ~/.local/share/containers/oci/hooks.d/ ~/.local/share/containers/oci/hooks.bin/

Make podman hook.

cat <<EOF > ~/.local/share/containers/oci/hooks.d/testhook.json
{
  "version": "1.0.0",
  "hook": {
    "path": "${HOME}/.local/share/containers/oci/hooks.bin/oci-mount-fixer"
  },
  "when": {
    "always": true
  },
  "stages": ["createRuntime","prestart"]
}
EOF

Make executable to be ran by hook.

echo '#!/bin/bash
echo running mount-fixer
echo $0 > /var/home/core/.oci-mount-fixer
echo $@ >> /var/home/core/.oci-mount-fixer
echo env_below >> /var/home/core/.oci-mount-fixer
env >> ~/.oci-mount-fixer
' > ~/.local/share/containers/oci/hooks.bin/oci-mount-fixer

Ensure executable has execute perms.

chmod +x ~/.local/share/containers/oci/hooks.bin/oci-mount-fixer

Testing

So this hook executable isn't going to work, this is just something I'm using to collect data in testing. The data will be in ~/.oci-mount-fixer. Before beginning, I made sure prior tests are cleaned with the file not there. The file existing is how I know the execuable ran without enabling debug output.

[core@localhost ~]$ ls -lah ~/.oci-mount-fixer
ls: cannot access '/var/home/core/.oci-mount-fixer': No such file or directory

I run podman with my custom hooks dir. At this point the hook is working. GREAT!

[core@localhost ~]$ podman --hooks-dir ~/.local/share/containers/oci/hooks.d/ run --rm alpine sh -c 'echo hello world'
Error: OCI runtime error: error executing hook `/var/home/core/.local/share/containers/oci/hooks.bin/oci-mount-fixer` (exit code: 1)

[core@localhost ~]$ ls -lah ~/.oci-mount-fixer
-rw-r--r--. 1 core core 80 Dec 18 16:58 /var/home/core/.oci-mount-fixer

Clean up and check if it runs when adding volumes? This does not work. Podman is stating before running hooks. Not sure this is right.

[core@localhost ~]$ rm -rf ~/.oci-mount-fixer

[core@localhost ~]$ ls -lah ~/.oci-mount-fixer
ls: cannot access '/var/home/core/.oci-mount-fixer': No such file or directory

[core@localhost ~]$ podman --hooks-dir ~/.local/share/containers/oci/hooks.d/ run -v /does/not/exist:/usr/src --rm alpine sh -c 'echo hello world'
Error: statfs /does/not/exist: no such file or directory

[core@localhost ~]$ ls -lah ~/.oci-mount-fixer
ls: cannot access '/var/home/core/.oci-mount-fixer': No such file or directory

[flouthoc]

@protosam Above PR should remove early failure on stat and instead failure will be relayed from OCI runtime. But I am afraid crun would still fail before calling hooks.

May i ask which OCI runtime are you using and are you sure hooks are right thing for this use case ?

[ghost]

The podman machine is using crun. Here is the complete output of podman info, sorry it was not in my initial post.

My goal is to mount my network filesystem that is in development as needed to bridge host to guest VM. So I think that hooks are the right thing (though if I'm wrong, I'm open to being told that, my ego is smaller than my desire to have a fusefs that can handle named pipes and unix-sockets for me).

When reading oci-hooks.5.md, it leads me to believe that it should be capable of doing mount related tricks.

Getting this to work helps in solving issues that are trying to achieve similar results without having to patch podman for every filesystem that comes along. Issues like #8016 as well as pull requests like #12584 and #11454.

[flouthoc]

@protosam Above PR #12681 fixes this but we also need a tweak in crun which is being discussed here: containers/crun#827

[giuseppe]

I am not sure this should happen in the OCI runtime, storage is usually managed before the OCI runtime runs.

Can't the storage be mounted before running Podman?

[flouthoc]

After discussing with @giuseppe I agree hooks is not a good solution for this use-case.

One problem is nothing defines if createContainer hooks should be triggered before setting up mounts instead runtime-spec explicitly states that it should be done after creating runtime environment and before pivot_root: https://github.com/opencontainers/runtime-spec/blob/main/config.md#createcontainer-hooks where creating runtime environment should include mounts as well.

@protosam You could easily do that before invoking podman a small plumbing should get it going.

[rhatdan]

You can continue to discuss here.

[ghost]

The example here seems to illustrate mount related tasks being done. That seems to hint to me that it should be capable of doing whatever it needs to do without being impeded.

Regarding plumbing, I am not sure what you had in mind @flouthoc. The only thing I can imagine is wraping podman with a shell script to do something before running. This changes the binary behavior in a way that will have unexpected results when a user inevitably does something I never thought about, that works the unwrapped binary.

On another note, when asked if hooks were right for this, it led me to do some more reading.

What seems like a better solution is actually volume plugins. There are many examples that already exist and they can live inside of a container as well. A real show stopper with it is that there appears to be no way to set a different volume driver as the default, overriding the local driver. (Also noticed that podman create/run both lack the --volume-driver flag).

The plugins system has a ton of untapped potential. If there was a way to make that work, it seems quantifiably better than hooks would be.

Edit: Something that was confusing for me when initially reading about volume plugins was the terms "volume driver" and "storage driver". So far as I can tell "storage driver" is unrelated to volumes and refers to the container's underlying storage such as overlayfs.