podman: Invalid cross-device link with named volume

[nivekuil]

Describe the bug
Running a container in podman on fcos next, fails with an exdev error when it tries to rename a file mounted under a named volume to something in the same directory.

Reproduction steps
Steps to reproduce the behavior:

1. sudo podman run -v "redpanda-test:/var/lib/redpanda/data" docker.io/vectorized/redpanda:v21.11.12
2. the first run, it will start ok, you should see a log line with "Successfully started Redpanda!"
3. the second run, the container exits with Failure during startup: std::__1::__fs::filesystem::filesystem_error (error system:18, filesystem error: rename failed: Invalid cross-device link [/var/lib/redpanda/data/redpanda/kvstore/0_0/snapshot.partial.1649900837775.a1ZO] [/var/lib/redpanda/data/redpanda/kvstore/0_0/snapshot])

Expected behavior
Container should restart without issue, being able to rename the file.

Actual behavior
Container dies with an odd error

System details

• Bare Metal/QEMU/AWS/GCP/etc.
  Proxmox VM

• Fedora CoreOS version
  next (v 36.20220410.1.1)

Ignition config
very sparse, just ssh keys, networkmanager configs (would be sanitized) and podman.socket.

Additional information
The container starts without issue if I point the directory to a bind mount instead of a named volume. I can only reproduce this on fcos, not on Arch Linux with podman 4.0.3, kernel 5.17.1 or centos 8 stream with podman 4.0.2.

[dustymabe]

Thanks for the simple reproducer. I was able to reproduce.

This might be something that we need to engage the podman team on. For example I see containers/podman#13123 which touches on this particular failure.

Maybe you're using fuse-overlayfs on those other systems where you'd be using overlay on FCOS?

[nivekuil]

Maybe you're using fuse-overlayfs on those other systems where you'd be using overlay on FCOS?

nope, no output from sudo podman info --debug | grep fuse-overlayfs. I don't think overlay is involved here because that's just for the container filesystem, and a named volume is something else? There's definitely too much magic going on here for me to understand.

[dustymabe]

asked @giuseppe to look into this..

13:18:21  giuseppe | I think it is related to xfs quotas
13:18:33  giuseppe | the source file is created from a different project


14:13:50  giuseppe | yes, I confirm that
14:14:11  giuseppe | if I reset the projectid for each parent directory of /sysroot/ost
                   | ree/deploy/fedora-coreos/var/lib/containers/storage/volumes/redpan
                   | da-test/_data/redpanda/kvstore/0_0/snapshot.partial.1649955476746.
                   | HdmP then it works
14:21:22  giuseppe | on the other hand, redpanda could handle EXDEV in a nicer way and
                   | copy the file then rm the old one

he's still investigating..

[nivekuil]

I also see this error when trying to set up ScyllaDB so it's probably coming from their shared library, seastar. I doubt whatever seastar API making the rename syscall will ever behave like mv because the library's purpose is predictable performance. Thanks for looking into it

[giuseppe]

@rhatdan do you think we need a mechanism to "relabel" volumes with the container quota projid?

The issue is that a file in a directory owned by a different projid cannot be linked or renamed causing link(2) and rename(2) to fail with EXDEV.

[dustymabe]

This issue is worked around upstream in containers/podman@d0b96a5

The workaround is to create the volume before you create the container(s) and specify -o o=noquota:

podman volume create -o o=noquota $ID

[dustymabe]

It appears that commit made it into podman v.4.2.0.

[dustymabe]

The fix for this went into next stream release 36.20220820.1.0.

[dustymabe]

The fix for this went into testing stream release 36.20220820.2.0.

[dustymabe]

The fix for this went into stable stream release 36.20220820.3.0.