No network inside container

[ptomato]

The containers I create with toolbox (0.0.15, with podman 1.6.1, Fedora 30) seem to have no network access.

~$ toolbox enter -r 31
⬢~$ sudo dnf install ninja
Fedora Modular 31 - x86_64                      0.0  B/s |   0  B     00:00    
Failed to download metadata for repo 'fedora-modular'
Error: Failed to download metadata for repo 'fedora-modular'

Looking at the debugging info requested on #152 I see it is because the host's /etc/resolv.conf
is a symlink not accessible inside the container:

⬢~$ ls -l /etc/hosts
lrwxrwxrwx. 1 root root 19 Oct 20 16:13 /etc/hosts -> /run/host/etc/hosts
⬢~$ ls -l /run/host/etc/hosts
-rw-r--r--. 1 nobody nobody 158 Apr  9  2019 /run/host/etc/hosts
⬢~$ ls -l /etc/resolv.conf
lrwxrwxrwx. 1 root root 25 Oct 20 16:13 /etc/resolv.conf -> /run/host/etc/resolv.conf
⬢~$ ls -l /run/host/etc/resolv.conf 
lrwxrwxrwx. 1 nobody nobody 35 Oct 20 15:09 /run/host/etc/resolv.conf -> /var/run/NetworkManager/resolv.conf
⬢~$ ls -l /var/run/NetworkManager/resolv.conf
ls: cannot access '/var/run/NetworkManager/resolv.conf': No such file or directory

Also, not sure if it's related, in the system journal this message appears when trying to access the network:

dbus-broker[1052]: A security policy denied :1.2347 to send method call /org/freedesktop/login1:org.freedesktop.login1.Manager.CreateSession to org.freedesktop.login1.

This seems similar to #187 but I don't think it's a duplicate. I have Flatpak 1.4.3 which according to that issue should include the necessary bugfix.

[debarshiray]

Thank you very much for the detailed report!

This seems similar to #187 but I don't think it's a duplicate.

Yes, indeed.

I have Flatpak 1.4.3
which according to that issue should include the necessary bugfix.

It's a bit more convoluted.

Like Flatpak containers, we also want Toolbox containers to keep some files synchronized with the host, and you can do that in multiple different ways.

One low-tech option is to bind mount /etc from the host somewhere inside the container (eg., /run/host/etc) and then establish symbolic links inside the container. This is what we currently do for /etc/resolv.conf. However, that doesn't work if the file on the host is itself a symbolic link to somewhere outside /etc, which is the situation in this case.

Another option is to use flatpak-session-helper to keep a clone of the files with inotify on the host and then bind the copies into the container. That should sniff out all the symbolic links. We already do this for timezones, but would be nice to have flatpak/flatpak#2916 in place before we can adopt it more widely (eg., for /etc/resolv.conf). Hence the talk about Flatpak 1.4.0.

Except, now, we are trying to get sudo toolbox ... working to cater to CoreOS' use-cases. On CoreOS, people want to use Toolbox containers to install random tools to debug the host OS, for which you need containers created with rootful Podman. When running under root, the lack of a D-Bus session/user instance becomes an obstacle for the flatpak-session-helper approach.

Therefore, I am now thinking that we should use realpath(1) to resolve the host's symbolic links and bind mount more locations and move away from flatpak-session-helper.

Sorry about all this back and forth on this issue. :/

Anyway, is your host (is it Endless OS?) really set up to have a symbolic link to /run/NetworkManager/resolv.conf? Or did you override the default handling of resolv.conf by creating the symbolic link yourself? (See in the NetworkManager.conf(5) manual.) Apologies in advance if this is a somewhat stupid question - my understanding of NM's handling of /etc/resolv.conf is quite fresh.

Finally, I think the log from dbus-broker about the method call is likely unrelated.

[ptomato]

Anyway, is your host (is it Endless OS?) really set up to have a symbolic link to /run/NetworkManager/resolv.conf? Or did you override the default handling of resolv.conf by creating the symbolic link yourself? (See in the NetworkManager.conf(5) manual.)

The host is Fedora 30 and it is really set up this way. I didn't make that change myself so I don't know how that happened or whether it was the default. (It's been upgraded steadily from at least Fedora 26 and possibly earlier, so maybe it was some default in an older Fedora version.)

[debarshiray]

The host is Fedora 30 and it is really set up this way. I didn't make that
change myself so I don't know how that happened or whether it was the
default. (It's been upgraded steadily from at least Fedora 26 and possibly
earlier, so maybe it was some default in an older Fedora version.)

That sounds a lot like #290 which is another duplicate of #187

I wonder if there is/was a bug in NetworkManager or Fedora which somehow turns /etc/resolv.conf into a symbolic link during an update. I always do fresh installs, and I checked everything from Fedora 29 to 31, both Silverblue and Workstation, and don't see any symbolic links.

For the time being, you could remove the symlink, copy /run/NetworkManager/resolv.conf to /etc/resolv.conf as a regular file, and verify that rc-manager in NetworkManager.conf is set to symlink. The NetworkManager.conf(5) manual lists the default value, because it can depend on NM's build options.

Maybe it's worth filing a bug against NM to explore the possibility of an upgrade bug?

[debarshiray]

Duplicate of #187

[ptomato]

See https://gitlab.freedesktop.org/NetworkManager/NetworkManager/issues/269

[debarshiray]

Thanks for pursuing this, @ptomato

[ptomato]

For completeness, sudo rm /etc/resolv.conf and a reboot fixed the issue for me.