multipass does not look for new IP addresses

[sinclert-canonical]

Describe the bug
Multipass 1.14.1 graphical UI freezes upon the installation of the network-manager snap, on an Ubuntu 24.04.1 VM.

To Reproduce
The bug was experienced while interacting with the graphical interface, but it seems the CLI may be affected too, as trying to execute multipass info after having the graphical UI frozen, does not work either.

To reproduce it via CLI:

multipass launch --cpus 2 --memory 4G --disk 5G --name Testing 24.04
multipass exec Testing -- sudo snap install network-manager

Expected behavior
The graphical UI would continue to respond to user input.

Logs

2024-11-11 12:32:45.899089 INFO Sent FindRequest{showImages: true, showBlueprints: false}
2024-11-11 12:32:45.921208 INFO Sent VersionRequest{}
2024-11-11 12:32:45.994749 INFO FindRequest{showImages: true, showBlueprints: false} received FindReply{showImages: true, imagesInfo: [{os: Ubuntu, release: 20.04 LTS, version: 20240821, aliasesInfo: [{alias: 20.04}, {alias: f}, {alias: focal}], codename: Focal Fossa}, {os: Ubuntu, release: 22.04 LTS, version: 20241002, aliasesInfo: [{alias: 22.04}, {alias: j}, {alias: jammy}], codename: Jammy Jellyfish}, {os: Ubuntu, release: 24.04 LTS, version: 20241106, aliasesInfo: [{alias: 24.04}, {alias: n}, {alias: noble}, {alias: lts}, {alias: default}], codename: Noble Numbat}]}
2024-11-11 12:32:45.994875 INFO FindRequest{showImages: true, showBlueprints: false} is done
2024-11-11 12:32:45.995144 INFO VersionRequest{} received VersionReply{version: 1.14.1+mac, updateInfo: {}}
2024-11-11 12:32:45.995273 INFO VersionRequest{} is done
2024-11-11 12:33:06.201736 INFO Sent LaunchRequest{instanceName: Testing, image: 24.04, numCores: 2, memSize: 4GB, diskSpace: 5GB}
2024-11-11 12:33:28.596499 INFO LaunchRequest{instanceName: Testing, image: 24.04, numCores: 2, memSize: 4GB, diskSpace: 5GB} is done
2024-11-11 12:33:28.859888 INFO Sent SSHInfoRequest{instanceName: [Testing]}
2024-11-11 12:33:28.874732 INFO SSHInfoRequest{instanceName: [Testing]} received SSHInfoReply{sshInfo: {Testing: {port: 22, privKeyBase64: *hidden*, host: 192.168.64.18, username: ubuntu}}}
2024-11-11 12:34:18.052272 ERROR Error on polling info
	gRPC Error (code: 9, codeName: FAILED_PRECONDITION, message: [ssh proc] failed to open session channel: '', details: [], rawResponse: null, trailers: {})

Additional info

• Host machine OS: macOS 15.1 (latest)
• Host machine arch: arm64.
• Multipass: 1.14.1 (latest)
• Multipass VM: Ubuntu 24.04.1 (latest)

Additional context

[sharder996]

Hi @sinclert-canonical, I did some debugging and I think I've found some hints to what the issue is and a way to work around it in the meantime.

First off, this seems to only happen when installing network-manager via the snap. Installing via apt does not break network configuration.

So, installing the network-manager snap reconfigures networking in the instance and causes the dhcp server on the host to issue a new IP address to instance. You can see this new IP address in the output of arp -an (it takes some time for the arp cache to timeout and the arp table to repopulate). Notice "incomplete" entry in output below:

arp -an                                                          
? (192.168.64.31) at (incomplete) on bridge100 ifscope [bridge]
? (192.168.64.32) at 52:54:0:d9:52:ae on bridge100 ifscope [bridge]
? (192.168.64.33) at 52:54:0:b2:2c:bd on bridge100 ifscope [bridge]
...

Multipass uses arp internally on MacOS to get IP addresses of instances, so an incorrect filtering of this output is causing Multipass to use the wrong IP. A current workaround that I've found at the moment is to simply restart the Multipass daemon.

sudo launchctl unload /Library/LaunchDaemons/com.canonical.multipassd.plist
sudo launchctl load /Library/LaunchDaemons/com.canonical.multipassd.plist

Multipass will pick up the new IP address and access should be restored to the instance.

[sinclert-canonical]

Hey @sharder996, thanks for the follow-up.

Indeed, installing network manager via apt instead of snap is the work-around that I followed when trying to install a VPN into my Multipass VM (see Canonical Discourse post). I will refer to the Multipass daemon reset work-around, if I ever need to install network-manager as a snap.

Curious though: is this problem not going to be addressed as long as a work-around exists? I am perfectly fine with whatever you folks decide, I just want to know whether to close the issue, or leave it open.

[ricab]

Hey @sharder996, do you have more details on what exactly is wrong with the filtering? And any ideas to select only the most recent IP?

Tagging @georgeliao for info.

[sharder996]

@ricab In the sample output from arp -an I showed in my previous post, you can see an entry with "(Incomplete)" instead of a MAC address. The entry right below it (192.168.64.32) has the same MAC address as the Multipass virtual machine we lost contact with. So, we would need to somehow flush the arp cache and force repopulate the table, and then get the IP based on the MAC address.

EDIT: Since Multipass has already gotten an IP address for the instance in the past, it doesn't look for a new one. We need to detect that the IP is stale/doesn't work and then search for a new one.

[georgeliao]

@ricab In the sample output from arp -an I showed in my previous post, you can see an entry with "(Incomplete)" instead of a MAC address. The entry right below it (192.168.64.32) has the same MAC address as the Multipass virtual machine we lost contact with. So, we would need to somehow flush the arp cache and force repopulate the table, and then get the IP based on the MAC address.

@sharder996 Thanks for the investigation and the explanation on this. The incomplete mac address issue is something occurred to me once while I was developing and testing that feature and I could not reproduce that after. It looks like the installing network-manager via snap is a deterministic way of making this happen. Anyway, I will look into the details later.

EDIT: Since Multipass has already gotten an IP address for the instance in the past, it doesn't look for a new one. We need to detect that the IP is stale/doesn't work and then search for a new one.

That explains why daemon restart helped.