Alternative install for Fedora

[abitrolly]

What are you trying to do?

multipass still doesn't work on Fedora 36 because of some snap limitations in #1448 that can not be fixed.

What's your proposed solution?

Provide an alternative installation method. Either .rpm or Homebrew for Linux.

[luis4a0]

Hi @abitrolly, we won't unfortunately support another packaging system. This is not because of the work which it would imply, but for the time its maintenance would take.

[townsend2010]

Also, it's not a snap limitation, it's that firewalld is running by default in Fedora. No matter what packaging we use, this issue will be problematic.

[abitrolly]

No matter what packaging we use, this issue will be problematic.

@townsend2010 all other packages, native .rpm and Homebrew installed do not have problems with the firewalld.

[abitrolly]

@luis4a0 how hard it would be to maintain Linux version of this Mac OS recipe https://github.com/Homebrew/homebrew-cask/blob/master/Casks/multipass.rb ?

[luis4a0]

Hi @abitrolly, I don't see big blockers (although I'm not an expert on Homebrew). If you want to try to do an unofficial install mechanism, it would be a nice addition to complement packaging systems we provide. Thanks!

[abitrolly]

@luis4a0 Homebrew looks like the official install mechanism for Mac OS, so why not just extend it to Linux?

[luis4a0]

In understand your point, but implementing something also incurs in mainenance costs and we already have one install method for Linux. It is hard to tell if this will have place in the features we decide to implement in the near future.

[abitrolly]

@luis4a0 but that's the point in having the official install method if it doesn't work?

[townsend2010]

Hi @abitrolly,

There are a few things going on here. As of today, it doesn't matter what packaging type we use for Multipass, Multipass simply does not intergrate firewalld support in it and this is needed to allow the mpqemubr0 bridge to work with the Fedora host. That said, if we did integrate with firewalld like what libvirt does, then we'd need to explore if there are any issues with strict Snap confinement and firewalld.

Also, regarding Homebrew support on Linux, we (the Multipass team) do not do the Homebrew Multipass recipe on macOS. This is done by a gracious community member. If Homebrew support on macOS began to lag, then we'd need to evaluate mentioning it as a supported install method or maintain it ourselves. Also, the macOS Homebrew recipe basically downloads the official macOS Multipass package and then extracts it in Homebrew specific paths and has uninstall and zap specific directives. I don't think this easily translates to downloading and extracting a Snap into Homebrew specific paths.

[abitrolly]

Multipass simply does not intergrate firewalld support in it and this is needed to allow the mpqemubr0 bridge to work with the Fedora host.

Why it needs mpqemubr0 bridge? Why it is not possible to access the shell in VM without firewall?

macOS Homebrew recipe basically downloads the official macOS Multipass package and then extracts it in Homebrew specific paths and has uninstall and zap specific directives. I don't think this easily translates to downloading and extracting a Snap into Homebrew specific paths.

I see. Should be mentioned that Homebrew is unofficial. Otherwise just doing the same compilation that is made for macOS, but targeting Linux should produce the binaries that are probably will end up in the same location. Without additional complications by snap packaging, which doesn't work on Fedora anyway.

[townsend2010]

Why it needs mpqemubr0 bridge? Why it is not possible to access the shell in VM without firewall?

This is what is needed for the VM's to reach the Internet behind a NAT'd bridge and for users to shell/exec into the instance. Currently, we only support reaching Multipass instances via SSH by this network. We do have plans to some day be able to log into the instances via the console, but I'm not sure how useful an instance would be without some sort of network access.

Should be mentioned that Homebrew is unofficial.

Why? It uses the official package "under the hood" meaning it uses the .pkg package that we produce and sign. We should promote community members who take the time to do things to improve Multipass and by referring to the Homebrew package, we are doing just that.

Otherwise just doing the same compilation that is made for macOS, but targeting Linux should produce the binaries that are probably will end up in the same location.

Again, it's the official macOS Multipass package found at (as of this writing) https://github.com/canonical/multipass/releases/download/v1.9.1/multipass-1.9.1+mac-Darwin.pkg that Homebrew uses. The Homebrew cask does not compile anything. It downloads that package and extracts it. Packaging for macOS is much different than packaging for Linux, so it's not so simple to adapt.

Without additional complications by snap packaging, which doesn't work on Fedora anyway.

Are there other non-firewalld issues in Fedora related to the Multipass Snap package that keep it from working?

[abitrolly]

This is what is needed for the VM's to reach the Internet behind a NAT'd bridge and for users to shell/exec into the instance.

I don't get it. VM is just a user space app, unlike containers, which need to interact with Linux kernel. There are other user level snaps on my Fedora like heroku and zoom-client that are able to reach the internet just fine.

[townsend2010]

A VM (virtual machine) is not just a user space app, it's an operating system running in a virtual computer. Since it's an operating system, it needs it's own virtual network interface and the host needs some way to route traffic from this virtual network adapter to the host's own network and then on out to the internet. A user level app as you suggest just opens a TCP socket to some server on the Internet and a way it goes. But virtual machines are much, much different and networking is much more complicated. It's just a fact with virtual machines no matter the hypervisor being used.

[abitrolly]

Since it's an operating system, it needs it's own virtual network interface and the host needs some way to route traffic from this virtual network adapter to the host's own network and then on out to the internet. A user level app as you suggest just opens a TCP socket to some server on the Internet and a way it goes.

Why QEMU can not do NAT in userspace? So that when VM tries to make outgoing TCP connection, the QEMU could wrap and forward it without VM even knowing that there is no real network adapter on the host.

Are there other non-firewalld issues in Fedora related to the Multipass Snap package that keep it from working?

I could not get past that firewalld (if it is firewalld) in #2564, so hard to tell. https://github.com/teemtee/tmt/ is able to run VMs for testing just fine, but it doesn't use snap packaging.

[townsend2010]

Why QEMU can not do NAT in userspace?

It can, but we found it to be very limiting and network speed was very poor compared to how we set up networking for Multipass instances.

I could not get past that firewalld (if it is firewalld) in #2564...

Did you add mpqemubr0 as a trusted firewalld zone as I pointed out in #1448 (comment)?

https://github.com/teemtee/tmt/ is able to run VMs for testing just fine

I see no evidence in the tmt source code that they call qemu directly. It looks like they possibly use libvirt and libvirt automatically sets up it's virtual bridge in a firewalld trusted zone. This is the part Multipass is missing, the firewalld integration.

[wheelerlaw]

I see no evidence in the tmt source code that they call qemu directly. It looks like they possibly use libvirt and libvirt automatically sets up it's virtual bridge in a firewalld trusted zone.

Is there a good reason why Multipass is calling out to qemu directly rather than using a library like libvirt?

[townsend2010]

Hi @wheelerlaw,

Multipass is able to use libvirt on Linux. See https://multipass.run/docs/set-up-the-driver#heading--linux-use-libvirt.

Hope this helps!

[townsend2010]

Also, to answer the question of:

Is there a good reason why Multipass is calling out to qemu directly..

We want to also provide an "all in one" solution where there are no dependencies on other libraries such as libvirt, lxd, etc. We want to give users some choices on what they prefer to use.