create volume path if not exists

[pruje]

FEATURE REQUEST

Description
When running a container with mounted volumes, if the directory does not exists, podman returns an error. This is very annoying when using a tons of volumes (often declared in podman-compose files), because we have to mkdir -p ... each path before running the container.

Steps to reproduce the issue:

docker run ... -v /path/to/directory:/path ...
Error: error checking path ...: no such file or directory

If /path/to/directory does not exists, podman should create it as an empty directory.
This is the way docker works.

Thanks!

[mheon]

@rhatdan I believe you had some thoughts on this one?

[rhatdan]

@pruje I believe this is a bug in Docker, and should return an error. We have decided to not create the source directory if it exists. The reason that I do not like this, is users could easily have a typo in the source path, which will lead to one or more directories being created on their system. If a user wants the source directory then they should create it. Not have it created as a side effect.

[domdomegg]

Summary

When running podman run or podman create commands using the -v option for a host volume that does not yet exist, the command fails with an error (either Error: error checking path ...: no such file or directory or Error: statfs ...: no such file or directory).

However, this is different to Docker, which will create the folder (When the host directory of a bind-mounted volume doesn’t exist, Docker automatically creates this directory on the host for you. In the example above, Docker creates the /doesnt/exist folder before starting your container.).

Many people come to Podman for its "Support for a Docker-compatible CLI interface", and get confused by this. However, the explicit decision by the containers team is that this is unwanted behaviour and podman should not copy it.

Suggestion

Motivation: This difference between podman and docker appears to have tripped up a lot of people. Especially where people are keen to get started using podman, this feels like a bad user experience to start off with. It also creates burden on the maintainers to keep having to deal with these duplicate tickets (across both podman but also other podman-related projects). I think we could improve podman to help users understand what is going wrong, and debug their problem. This hopefully will save them time, make them more likely to continue using podman, and result in less duplicate issues for maintainers.

Proposal: Detecting when an error is thrown because the file does not exist when trying to mount a volume. When this occurs, rather than printing out the raw error (which might not make it immediately obvious what has gone wrong), we print out some help text such as:

Failed to mount volume due to missing host file or directory: /doesnt/exist
Not creating directories automatically is a deliberate difference between Podman and Docker
(see https://github.com/containers/podman/blob/main/transfer.md#behavioural-differences-and-pitfalls)
If want this folder to be created, run: mkdir -p /doesnt/exist

(The last line shows mkdir -p for macOS/Linux, and mkdir for Windows. The exact wording can of course be tweaked. All other behaviour stays the same e.g. same exit code, program still exits)

[domdomegg]

(also so this doesn't get too lost, can we reopen this as a kind/feature @Luap99?)

[rhatdan]

We already commented on this, as we believe this is fundamentally a bug in Docker, that was never fixed. Creating a directory as a side effect of a mispelled directory is a bug, not a feature.

[domdomegg]

Thanks @rhatdan - my suggestion is not to change the behaviour to add this side effect. My suggestion is to print out a more helpful error message that can help users understand what to do next, increase the chance they stick with podman, and decrease unnecessary support queries: #6234 (comment)

[olemorud]

We already commented on this, as we believe this is fundamentally a bug in Docker, that was never fixed. Creating a directory as a side effect of a mispelled directory is a bug, not a feature.

This bug is sadly required by GitHub actions to start containers. Although creating a wrapper script to emulate this bug is somewhat trivial, it still adds an extra layer of complexity in the setup of self-hosted GitHub runners using Podman. Adding some way of emulating this bug would make rhel machines a much more viable option for GitHub CIs. Either way apologies for necroing a closed issue and thanks for all the amazing work that has been done on this project.

[rhatdan]

Sadly people are hard coding a broken behaviour into other tools, if GitHub actions are doing this with Podman support then it should be reported to them, if this is just podman-docker triggering this, then Github actions might push back.

[RanabirChakraborty]

while running podman it's showing Error: statfs /tmp/jcasc-managed.8dz: no such file or director do anyone know how can I fix it? Using in MacOS

[rhatdan]

touch /tmp/jcasc-managed.8dz

[RanabirChakraborty]

Actually, the file names are not fixed everytime I'm running podman, the file name is changing. As an example the last error was Error: statfs /tmp/jcasc-managed.4sU: no such file or directory

[rhatdan]

Who is creating these names and why are they volumes and not just created in the image.

[TheNerdMan]

We already commented on this, as we believe this is fundamentally a bug in Docker, that was never fixed. Creating a directory as a side effect of a mispelled directory is a bug, not a feature.

While I totally get Podman making the decision to fix this bug, for compatibility it can break some things. Could there be an option of a flag when running podman run or podman create that creates directories if the they don't exist?

I feels like creating a flag for those commands satisfies the argument of directories shouldn't be created as a side effect. While still giving users the ability to set up mounted volumes

[rhatdan]

What is the difference between doing
mkdir -p foobar; podman run -v ./foobar:/var/foobar ...
Versus
podman run --mkdir-volumes-if-they-dont-exist -v ./foobar:/var/foobar ...

[pylight]

The difference is the folder permissions and owner/group rights.

I would therefore say it's maybe not a bug in docker but a feature. Because automatically created host folders will have the correct folder permissions (owner/group from the user within the container). This is usually not working if the folder was created on the host machine with "mkdir" command. I experienced this problem as well when switching from docker to podman and trying to run wg-easy container (see docker run command here: https://github.com/wg-easy/wg-easy). The container will run with docker but it will unfortunately fail with podman if you don't create an ~/.wg-easy folder with the correct UID/GID of the user that is used inside the container.

[rhatdan]

You can add a :O to the volume mount to set the owneship correct for the volume.

[cmhull42]

On what basis are we saying this is a bug in docker? Not everybody is trying to put some existing data into a container, what if my intention is to mount a persistent work directory into the container that should rightfully be initialized empty? Yes I can just manage the lifecycle of that myself, but A), why should I worry about state when I don't have to and B) why deliberately introduce incompatibilities with docker? It's not such a huge feature either way, but I'd think we'd certainly bias towards not breaking stuff.

[rhatdan]

I follow the linux standard of mount

$ sudo mount -o bind /tmp/bogus /tmp/foobar
Place your right index finger on the fingerprint reader
mount: /tmp/foobar: mount point does not exist.
       dmesg(1) may have more information after failed mount system call.