Issue with podman machine init and option --volume on macOS (virtfs support is disabled, says qemu)

[fc7]

/kind bug

(4.0.0-dev version)

Description

I tested the current 4.0.0-dev version on a Mac using brew install --head podman (after unlinking the 3.4.4 version).
My objective was to try the forthcoming feature with the option --volume. Unfortunately in so doing I encountered the following error:

% podman machine start experimental                            
Starting machine "experimental"
INFO[0000] waiting for clients...                       
INFO[0000] new connection from  to /var/folders/6x/0bjf97xs2210ll8zxggh1v040000gn/T/podman/qemu_experimental.sock 
Waiting for VM ...
qemu-system-x86_64: -virtfs local,path=/Users/fch,mount_tag=vol0,security_model=mapped-xattr: There is no option group 'virtfs'
qemu-system-x86_64: -virtfs local,path=/Users/fch,mount_tag=vol0,security_model=mapped-xattr: virtfs support is disabled
Error: dial unix /var/folders/6x/0bjf97xs2210ll8zxggh1v040000gn/T/podman/experimental_ready.sock: connect: no such file or directory
ERRO[0003] cannot receive packets from , disconnecting: cannot read size from socket: EOF 
ERRO[0003] cannot read size from socket: EOF 

Question: how can I enable virtfs support in qemu? I have the latest stable version of qemu installed via brew:

% brew info qemu  
qemu: stable 6.2.0 (bottled), HEAD

Or is there maybe another supported driver than virtfs for macOS that can be passed to --volume-driver?

Steps to reproduce the issue:

1. Install dev version of podman on macOS with brew install --head podman
2. Run podman machine init experimental --volume /Users/foo:/mnt/foo
3. Run podman machine start experimental

Output of podman version:

podman version 4.0.0-dev

Output of podman info --debug:

Cannot be executed due to above issue. However when creating and starting a different machine without the --volume option, I get this error (though this might be unrelated to the above and can be ignored for the moment):

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: Connection to bastion host (ssh://core@localhost:63038/run/user/1000/podman/podman.sock) failed.: dial tcp [::1]:63038: connect: connection refused

Any help appreciated on this!

[afbjorklund]

See #8016 (comment)

• Support native source folder for volume mount in remote model #8016

Formula: https://github.com/afbjorklund/homebrew-core/blob/qemu-9p-darwin/Formula/qemu.rb

EDIT: Update link to patch (.diff and not .patch), and use the branches from willcohen

[afbjorklund]

Or is there maybe another supported driver than virtfs for macOS that can be passed to --volume-driver?

There are no other drivers at the moment. I planned on doing one for sshocker, but it was better to run as a command.

Theoretically one could add drivers for NFS and for SMB, but I don't think anyone has volunteered to do it yet.

Eventually we are hoping for a virtio-fs driver for darwin (using vsock), but that could be well in the future (no time plan).

---

EDIT: On the Podman community cabal, it was revealed that there might be virtio-fs for darwin already this year (2022)

The engineers are busy rewriting it in Rust, but after that it should be possible to use it with Podman 4.x something

[fc7]

Thanks a lot @afbjorklund that looks promising. I will try that formula and report back.

[fc7]

@afbjorklund I get this issue with your patch when installing your formula for qemu with virtfs (build-from-source):

patching file hw/9pfs/9p-util-darwin.c
can't find file to patch at input line 818
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/hw/9pfs/9p-util-linux.c b/hw/9pfs/9p-util-linux.c
|index defa3a413075..3902378e627c 100644
|--- a/hw/9pfs/9p-util-linux.c
|+++ b/hw/9pfs/9p-util-linux.c
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored
patching file hw/9pfs/9p-util.h
patching file hw/9pfs/9p.c
Hunk #1 succeeded at 27 (offset 5 lines).
patching file hw/9pfs/9p-local.c
Hunk #1 succeeded at 673 (offset 5 lines).
Hunk #2 succeeded at 688 (offset 5 lines).
Hunk #3 succeeded at 701 (offset 5 lines).
patching file hw/9pfs/9p-util-darwin.c
can't find file to patch at input line 960
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/hw/9pfs/9p-util-linux.c b/hw/9pfs/9p-util-linux.c
|index 3902378e627c..06399c59199c 100644
|--- a/hw/9pfs/9p-util-linux.c
|+++ b/hw/9pfs/9p-util-linux.c
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored

I did not have the time to investigate in details but at first sight the files to patch should be present in the upstream branch: https://gitlab.com/wwcohen/qemu/-/tree/9p-darwin/hw/9pfs but there were some commits a week ago, so could it be that the patch is no longer compatible? Should I try to use the upstream branch on gitlab without your patch?

[fc7]

Oh thanks! I am building from git right now using branch v6.2.0-9p-darwin (brew install --head <your-formula-without-the-patch> ... let's see how it goes. I'll keep your diffs as a fallback.

[afbjorklund]

Any misapplies are all mine, I just copied from git-branches-on-gitlab and patches-on-mailing-list over to regular qemu github...

The latest patches (for qemu 7.0) are linked here, think it's up to "v7" now: NixOS/nixpkgs#122420 (originals were from 2018)

[willcohen]

Latest version of the backport branch, updated to the latest v7 patch set, is here: https://gitlab.com/wwcohen/qemu/-/commits/v6.2.0-9p-darwin
Still running the build. If it works okay I'll get it over to a github repo so that you can generate the patch with one URL.

Note that this still isn't to be considered 100% final. The branch is still subject to change pending any remaining feedback during the upstreaming process.

Edit: Patch for the v7 patchset is here: https://github.com/willcohen/qemu/compare/v6.2.0..0024dfc24f88410fe9d85ef8e4a27cbc7283b87a.patch

I'll update my branch at https://github.com/willcohen/qemu/tree/v6.2.0-9p-darwin with subsequent versions (v8, etc) as they happen.

[afbjorklund]

Thanks! I changed my brew formula to use your backport branch, v6.2.0-9p-darwin.

qemu/qemu@v6.2.0...willcohen:0024dfc24f88410fe9d85ef8e4a27cbc7283b87a

It still builds an insane number of architectures, so takes "a while" and is a 190M tgz...

🍺  /home/linuxbrew/.linuxbrew/Cellar/qemu/6.2.0: 209 files, 952.3MB, built in 14 minutes 54 seconds

[fc7]

Thansk @willcohen ! I was successful in building your branch v6.2.0-9p-darwin this morning, using a modifed brew formula (without any patch). (I had a compile error yesterday regarding an undeclared symbol but after a new pull this morning it went away, which is consistent with your comment above).
Same experience as @afbjorklund regarding build time (worse actually, as I am on an older Intel Macbook).

[fc7]

I forgot the most important: with podman 4.0.0-dev (also rebuilt this morning) the --volume option works like a charm. But before closing this, I should point out that the podman documentation is only describing the latest unreleased version, which can be confusing to many people, as more and more users are migrating from Docker to podman (at least this is the case in my company). It would be good to be able to switch the documentation from "latest" to the current stable version (3.x). At least it had the nice side-effect of my discovering the new --volume option, hence my attempt to use it and the above report. Thanks again for your help!

[afbjorklund]

The confusing documentation is a known issue:

• https://github.com/containers/podman.io/issues/485

[afbjorklund]

Found the issue with the patch, it wasn't renaming the file - which was an issue on darwin, but not on linux.

Fixed: https://github.com/afbjorklund/homebrew-core/blob/qemu-9p-darwin/Formula/qemu.rb

🍺  /opt/homebrew/Cellar/qemu/6.2.0: 162 files, 556.8MB, built in 4 minutes 35 seconds

Should also work with --HEAD now (https://gitlab.com/wwcohen/qemu.git), got the syntax of the patch fixed.

[willcohen]

Is this something I need to correct for v9?

[afbjorklund]

Is this something I need to correct for v9?

It was just how git creates patches, you need to use git am to apply them - simple GNU patch is not smart enough:

 fsdev/file-op-9p.h                     |  9 ++++++++-
 fsdev/meson.build                      |  1 +
 hw/9pfs/9p-local.c                     | 27 +++++++++++++++++++++------
 hw/9pfs/9p-proxy.c                     | 38 +++++++++++++++++++++++++++++++++++---
 hw/9pfs/9p-synth.c                     |  6 ++++++
 hw/9pfs/9p-util-darwin.c               | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 hw/9pfs/{9p-util.c => 9p-util-linux.c} |  2 +-
 hw/9pfs/9p-util.h                      | 35 +++++++++++++++++++++++++++++++++++
 hw/9pfs/9p.c                           | 42 ++++++++++++++++++++++++++++++++++++------
 hw/9pfs/9p.h                           | 18 ++++++++++++++++++
 hw/9pfs/codir.c                        |  4 +++-
 hw/9pfs/meson.build                    |  3 ++-
 include/qemu/osdep.h                   | 12 ++++++++++++
 include/qemu/xattr.h                   |  4 +++-
 meson.build                            | 15 +++++++++++----
 os-posix.c                             | 35 +++++++++++++++++++++++++++++++++++
 tests/qtest/virtio-9p-test.c           |  2 +-
 17 files changed, 292 insertions(+), 25 deletions(-)

==> Patching
==> Applying v6.2.0...willcohen:0024dfc24f88410fe9d85ef8e4a27cbc7283b87a.patch
patching file fsdev/file-op-9p.h
patching file hw/9pfs/9p-local.c
patching file hw/9pfs/9p.c
patching file include/qemu/xattr.h
patching file hw/9pfs/9p-util.c
patching file hw/9pfs/meson.build
patching file hw/9pfs/9p-proxy.c
patching file hw/9pfs/9p-synth.c
patching file hw/9pfs/9p.c
patching file hw/9pfs/9p-local.c
patching file hw/9pfs/9p-proxy.c
patching file hw/9pfs/9p-synth.c
patching file hw/9pfs/9p-util.h
patching file hw/9pfs/9p.c
patching file hw/9pfs/codir.c
patching file hw/9pfs/9p-util.h
patching file hw/9pfs/9p.c
patching file hw/9pfs/9p.c
patching file hw/9pfs/9p.h
patching file hw/9pfs/9p-util-darwin.c
patching file hw/9pfs/meson.build
patching file hw/9pfs/9p-local.c
patching file hw/9pfs/9p-util.h
patching file hw/9pfs/9p-local.c
patching file include/qemu/osdep.h
patching file meson.build
patching file os-posix.c
patching file tests/qtest/virtio-9p-test.c
patching file fsdev/meson.build
patching file meson.build

It seems like the failure to rename wasn't fatal on Linux, but did kill the build on Darwin ?

[afbjorklund]

Did get a build error with the latest version, not sure if I need to update something ?

==> Cloning https://gitlab.com/wwcohen/qemu.git
Updating /Users/m1/Library/Caches/Homebrew/qemu--git
==> Checking out branch 9p-darwin

Last 15 lines from /Users/m1/Library/Logs/Homebrew/qemu/02.make:
        return -ENOTSUPP;
                ^
1 error generated.

Error: Your Xcode (13.1) is outdated.
Please update to Xcode 13.2.1 (or delete it).
Xcode can be updated from the App Store.

---

../os-posix.c:348:10: warning: address of function 'pthread_fchdir_np' will always evaluate to 'true' [-Wpointer-bool-conversion]
    if (!pthread_fchdir_np) {
        ~^~~~~~~~~~~~~~~~~
../os-posix.c:348:10: note: prefix with the address-of operator to silence this warning
    if (!pthread_fchdir_np) {
         ^
         &
../os-posix.c:350:17: error: use of undeclared identifier 'ENOTSUPP'
        return -ENOTSUPP;
                ^

error_report_once("pthread_fchdir_np() is not available on this macOS version");

[willcohen]

Okay. Will check. The check for the function changed with v8 so I’ll make sure I didn’t mess anything up before I resubmit. Many thanks!

[willcohen]

Oh — quick first question. What version of macOS are you on?

[afbjorklund]

What version of macOS are you on?

% sw_vers 
ProductName:	macOS
ProductVersion:	12.2.1
BuildVersion:	21D62

Compiled (and ran) OK with workaround:

diff --git a/os-posix.c b/os-posix.c
index 84be7c74ce..11d04af4fa 100644
--- a/os-posix.c
+++ b/os-posix.c
@@ -347,7 +347,7 @@ int qemu_mknodat(int dirfd, const char *filename, mode_t mode, dev_t dev)
     int preserved_errno, err;
     if (!pthread_fchdir_np) {
         error_report_once("pthread_fchdir_np() is not available on this macOS version");
-        return -ENOTSUPP;
+        return -1; // ENOTSUPP;
     }
     if (pthread_fchdir_np(dirfd) < 0) {
         return -1;

🍺  /opt/homebrew/Cellar/qemu/HEAD-f6620d0: 160 files, 609.6MB, built in 4 minutes 34 seconds

But it's a lot smaller with: ../configure --target-list=aarch64-softmmu,x86_64-softmmu --enable-hvf --enable-virtfs

[willcohen]

Ah. It's a typo on master versus a patch version. It is supposed to be -ENOTSUP! Will fix when I submit v9.

[baude]

@willcohen was test driving the patch today ... can qemu do anything for us to help with the difference on uid/guid between the mac and the linux vm where the mount is made.

[willcohen]

I’ve seen this mentioned a few places but haven’t fully wrapped my head around it. To my knowledge the uid is just passed through directly. Is the idea here that qemu could reassign the number en route so it looks like one thing on the host and one thing on the guest? Is it always going to be predicable on the coreos side, so that hypothetically it’d be something we’d want to see invoked on the command line when starting up qemu?

[baude]

I believe it would be predictable but it is always nice to allow it to be made by entry.

[willcohen]

For existing qemu functionality, the closest thing I can find in https://wiki.qemu.org/Documentation/9psetup is to use -o access=<uid> when running mount on the Linux side. At first glance that seems like it should work, rather than relying on the uid being passed through from QEMU. I assume that that basically just ignores the macOS attributes being passed through. Has this been tried?

I submitted v9 today and hopefully there’s no more changes to it. If we need to sneak another flag into 9p on the qemu side we’ve got a week or two to put together another patch set before soft freeze!

[baude]

@willcohen would you mind providing a link to your v9 submission?

[willcohen]

Yep!

#8016 (comment)

https://lists.gnu.org/archive/html/qemu-devel/2022-02/msg05978.html

[afbjorklund]

I tested both the 6.2.0 version and the 7.0.0 (well, "to be") and besides the file rename in the patch and the typo they worked fine.

Fixing the qemu formula to cope with these changes was not so hard either, and after that it worked just like the linux version ?

    system "mv hw/9pfs/9p-util.c hw/9pfs/9p-util-linux.c" unless build.head?
    inreplace "os-posix.c", "-ENOTSUPP", "-ENOTSUP" if build.head?

I don't think there should be any more unrelated changes done to it, mapping uid and gid sounds like a separate new feature ?

It seemed perfectly predictable, the files were owned by nobody (or 501) in the VM and had nice xattr attached on the host:

% touch /tmp/podman/foo
% podman run -it -v /mnt/host:/mnt busybox touch /mnt/bar
 
% ls -l@ /tmp/podman/
total 0
-rw-------@ 1 m1  wheel  0 27 Feb 18:30 bar
        user.virtfs.gid 4
        user.virtfs.mode        2
        user.virtfs.uid 4
-rw-r--r--  1 m1  wheel  0 27 Feb 18:30 foo

[willcohen]

Okay. I’ll try to look a little more into this.

For what it’s worth, with the v9 patch the ENOTSUP typo shouldn’t be an issue anymore and I think you can remove it from the formula. The typo was in my older version of the branch on top of master, not in upstream HEAD itself.

[willcohen]

@afbjorklund do you know what's wrong with 9p-util-linux and why it's getting that error? I'm noticing it too when trying to build the backport branch on nix.

It seems like willcohen/qemu@c99f2b9 should be addressing it just fine.

[afbjorklund]

Sorry, which error ?

Patch didn't understand when both patching and renaming at the same time... So it only did the patching, but left the old file name. Had to help it.

Moral: never rename stuff

[willcohen]

The CONFIG_LINUX issue which prevents you from building without moving 9p-util.c to 9p-util-linux. It seems like there may be some other issue where QEMU is incorrectly setting CONFIG_LINUX on Mac with nix and brew, but I can't quite pin down what it is, and perhaps it was never relevant until this feature. Similarly, my WIP pr for nix has to remove the file from meson too.

[willcohen]

Sorry, which error ?

Patch didn't understand when both patching and renaming at the same time... So it only did the patching, but left the old file name. Had to help it.

Moral: never rename stuff

Right. Just to confirm, this should mean that it only matters for people running, say, git am on a generated patch backporting, right (and not a faux pas in the v9 patchset)? This won't be an issue for brew or nix once they're pulling 7.0?

[afbjorklund]

Right, normally it would build from a release tarball

But it is also some special to how brew runs patch, because if you run patch -p1 on the command-line it is fine:

patching file hw/9pfs/9p-util-linux.c (renamed from hw/9pfs/9p-util.c)

This was why I couldn't reproduce the build failures, had to try it myself using the actual brew command first...

[afbjorklund]

I didn't look closer at why it was renamed...

But you are right, it doesn't make sense since only $targetos should be defined at a time.

if test "$linux" = "yes" ; then
  echo "CONFIG_LINUX=y" >> $config_host_mak
fi

if test "$darwin" = "yes" ; then
  echo "CONFIG_DARWIN=y" >> $config_host_mak
fi

Unless Darwin defines __linux__ for some reason.

[afbjorklund]

@willcohen : it doesn't matter what the value is, the meson.build won't "compile" if the file is missing.

../hw/9pfs/meson.build:16:6: ERROR: File 9p-util-linux.c does not exist.

fs_ss.add(when: 'CONFIG_LINUXXX', if_true: files('9p-util-linux.c'))

[willcohen]

Got it -- then I'll tentatively assume that this is a backport-only issue affecting only those applying these generated patches, and it won't be an issue after 6.2.

[cron410]

It seems to be working now. I installed podman from Brew (my terminal history below)

brew install multipass
brew install podman
ln -s /usr/local/bin/podman /usr/local/bin/docker || true
sudo ln -s /usr/local/bin/podman /usr/local/bin/docker || true
podman machine init
podman machine start

but later had to tear it down to add the volume with:

  podman machine stop podman-machine-default
  podman machine rm podman-machine-default
  podman machine init -v /Users/username:/Users/username
  podman machine start

Initially the volume did not show on the first boot

...but rebooting the default generated coreos vm fixed it.

podman machine stop
podman machine start