Bridging to VPN interface and en0 removes SSH access

[fmoc]

I have a setup on a Mac mini M1 which is connected to the Internet via LAN (en0) and to a VPN interface (utun0 resp. bridge100). I am trying to set up networking without the NAT workaround mentioned here.

# first attempt using the bridge
> tart run ... --net-bridged bridge100
Error: no bridge interfaces matched "bridge100", available interfaces: en0 (or "Ethernet"), en1 (or "Wi-Fi")

# starts up, but doesn't allow me to SSH into the VM using the IP address provided by tart ip
> tart run ... --net-bridged utun0 --net-bridged en0

The latter is not an option, since I permanently need access via SSH to the VM for monitoring purposes.

How do I need to run the VM to have it connect to both networks as well as make sure I can access it from the host?

[fkorotkov]

Just to clarify:

1. You need VPN access inside your VM to access some private stuff.
2. You need local LAN bridge so you can SSH into the VM from a different host.

Did I understand your setup correctly? In that case I think we can an option to provide multiple interfaces for the bridged mode since VZVirtualMachineConfiguration supports it.

Can I also ask you to provide output of pfctl -s nat -a "com.apple.internet-sharing/shared_v4" command executed on the host where you run VMs.

[fmoc]

Yes, basically. Although I think the en0 bridge is used mostly for Internet/LAN access. I'm not sure what to bridge to to have access from the host

Can I also ask you to provide output of pfctl -s nat -a "com.apple.internet-sharing/shared_v4" command executed on the host where you run VMs.

Will do once I have one machine ready for testing again. Currently, they are busy (hosting 1-2 VMs using tart for CI purposes), so the NAT configuration is modified as described in the initial post.

In that case I think we can an option to provide multiple interfaces for the bridged mode since VZVirtualMachineConfiguration supports it.

Since --net-bridged can be specified more than once, I was assuming that this means we can already specify multiple interfaces to bridge to. Perhaps that is why the setup above is not working? You mentioned in the original bridge ticket you'd intend to support it, so I assumed that was already the case. If not, I think just allowing multiple bridged interfaces should fix the problem.

[edigaryev]

I'm not sure what to bridge to to have access from the host

Does the standard shared network (NAT) allows access from the host for you? Or I'm misunderstanding the notion of host?

The same shared network would also allow access to VPN resources, if the VPN routes are configured adequately. You can check this with pfctl -s nat -a "com.apple.internet-sharing/shared_v4", as @fkorotkov mentioned above.

[fkorotkov]

In the meantime I'm adding support for Bridged network to Orchard in cirruslabs/orchard#78.

In case of Orchard, just having one bridged network for VPN will be enough since SSHing is done through a runnel via an agent sitting on the host.

[fmoc]

Does the standard shared network (NAT) allows access from the host for you? Or I'm misunderstanding the notion of host?

Yes, that works fine. However, without the added NAT rule mentioned in #384 (comment), I cannot access the VPN interface utun*. The current rule set with the NAT rule (which I kind of need to hack in on demand, currently with a cronjob that runs every minute) looks like this:

> sudo pfctl -s nat -a com.apple.internet-sharing/shared_v4
No ALTQ support in kernel
ALTQ related functions disabled
nat on en0 inet from 192.168.64.0/24 to any -> (en0:0) extfilter ei
no nat on bridge100 inet from 192.168.64.1 to 192.168.64.0/24
nat on utun0 inet from 192.168.64.0/24 to any -> 172.XX.YY.ZZ  # VPN

The latter rule is the one I add using that cron job, which basically adds the following rule:

nat on utun0 from bridge100:network to any -> utun0

This rule is translated to the syntax seen above.

if the VPN routes are configured adequately

I think tart should probably receive a configuration feature or CLI parameter where you can specify the VPN interface so it can automatically add that rule. I'm sure I'm not the only person in this community that depends on such a setup.

[edigaryev]

I think tart should probably receive a configuration feature or CLI parameter where you can specify the VPN interface so it can automatically add that rule. I'm sure I'm not the only person in this community that depends on such a setup.

I'm afraid that this would be too much of a hack to implement since this requires root and messes up with Apple's PF tables.

I think a more productive direction would be figure out why in your case the NAT rules are not getting added automatically.

For example, when I run the Tailscale VPN, the needed PF rules are inserted automagically:

% sudo pfctl -s nat -a com.apple.internet-sharing/shared_v4 2>/dev/null | grep utun
nat on utun5 inet from 192.168.62.0/24 to any -> (utun5:0) extfilter ei

As a result, both LAN and VPN networks are accessible from the guests when running default shared (NAT) networking (no --net-softnet or --net-bridged).

Perhaps you could provide some ways to reproduce your case? And/or share some more details on your VPN configuration, like the VPN client used, the routes installed (netstat -rn -f inet | grep utun) and if the VPN connection is visible in Settings → Network?

[fmoc]

We use OpenVPN (directly, via homebrew, not via Tunnelblick).

> % sudo netstat -rn -f inet
Routing tables

Internet:
Destination        Gateway            Flags           Netif Expire
default            wan.1              UGScg             en0       
default            link#18            UCSIg       bridge100      !
10.90.4/24         v.p.n.1            UGSc            utun0       
10.90.5/24         v.p.n.1            UGSc            utun0       
127                127.0.0.1          UCS               lo0       
127.0.0.1          127.0.0.1          UH                lo0       
wan/26             link#6             UCS               en0      !
wan.1/32           link#6             UCS               en0      !
wan.1              8:5:e2:c8:93:ab    UHLWIir           en0    428
wan.7/32           link#6             UCS               en0      !
wan.7              14:98:77:84:46:57  UHLWIi            lo0       
169.254            link#6             UCS               en0      !
v.p                v.p.n.1            UGSc            utun0       
v.p.n/22           v.p.n.31           UGSc            utun0       
v.p.n.31           v.p.n.31           UH              utun0       
192.168.64         link#18            UC          bridge100      !
192.168.64.38      5a.32.5a.f9.53.5   UHLWIi      bridge100    498
192.168.64.39      fa.6f.af.8c.a4.bb  UHLWIi      bridge100    534
224.0.0/4          link#6             UmCS              en0      !
224.0.0.251        1:0:5e:0:0:fb      UHmLWI            en0       
255.255.255.255/32 link#6             UCS               en0      !

[edigaryev]

We use OpenVPN (directly, via homebrew, not via Tunnelblick).

Thanks for the details! I've actually tried playing with OpenVPN on macOS and manually setting the default gateway to point to be the OpenVPN, and I think my statement in #384 (comment) regarding the need for the default gateway is wrong.

The real cause seems to be whether the VPN client uses macOS APIs or configures the interface manually, e.g. the WireGuard seems to use the Network Extension, which results in the following Network screen:

And the NAT rules are added automatically, which is not the case for OpenVPN, which configures the VPN interface manually:

I've checked and the Tunnelblick seems to use the same underlying OpenVPN code, so no luck here.

[edigaryev]

Will check if allowing the tart run to take multiple --net-bridged flags (and thus adding multiple virtual Ethernet adapters to the guest) yields any positive results and will post my findings here.

[edigaryev]

I've just realized that --net-bridged utun* would never work, because it's not a bridged interface.

Neither manually creating a bridged interface with ifconfig would work, it simply won't be added to the VZBridgedNetworkInterface.networkInterfaces for some reason.

Unfortunately, at this point I don't see any solid options besides switching to a VPN solution that uses macOS APIs (e.g. WireGuard), which will guarantee proper NAT functioning.

As a side note, regarding the ability to specify multiple interfaces for a VM: it is possible to implement, but is somewhat complicated because we'd need to have more than one MAC-address for a VM.

If we re-use the single MAC from the VM's config, Virtualization.Framework will de-duplicate all the interfaces we pass to it internally based on MAC-address and the VM will run with only a single interface.

[fmoc]

Thanks! I would have never guessed that using some native macOS API to set up VPN in the UI, too, would make such a big difference...

I've just realized that --net-bridged utun* would never work, because it's not a bridged interface.

The interface doesn't have to be a bridge to be added to a bridge, right?

[edigaryev]

The interface doesn't have to be a bridge to be added to a bridge, right?

You're right, however, Virtualization.Framework's VZBridgedNetworkInterface only allows us to use "network interfaces available for bridging", which is unfortunate.

This definition seems to only include (1) physical interfaces like Wi-Fi and Ethernet, attachable adapters (e.g. USB to Ethernet) and (2) virtual interfaces created from the Apple's UI or using networking APIs (e.g. VPN connection).

[edigaryev]

Unfortunately, there's nothing we can do in the current state of things to resolve your issue.

Consider using a VPN solution with Network Extension support, for example, WireGuard.

[fmoc]

Yup, makes sense. Thanks for all the debugging, I hope this is going to help others, too.