Connections to the port exposed via --publish are dropped and do not reach the contained process #22959
Comments
Note that before enabling IPv6 support, the result was the same. |
|
Does something like this: work for you? |
|
First, changing ipv6_enabled to true doesn't do anything unless you actually add a ipv6 subnet to the config, compare a network created with Second, as rootless the default (podman) network isn't even used unless you specify |
|
Also as root we do support forward via ::1 at all, see #14491 |
|
@sbrivio-rh Nope: |
I don't really understand how public ips addresses are relevant here; I'm trying to connect to a container running on this same host; no networking is happening across [physical] hosts. |
|
Apparently my router was in some bogus state and had no public IPv6. I restarted it and now I have a public IPv6. The requirements is still a problem: sometimes I visit regions with no IPv6 connectivity, and I still want to run a container on my laptop and connect to it. Actually, sometimes I'm on a train with no public IPv6 or IPv4 at all. |
The configuration of the upstream interface is relevant because pasta, by default, tries to mimic as close as possible the host networking. By doing so, in the bigger picture, you can avoid NAT because the container inherits the addresses that are assigned to the upstream interface on the host. See also: #22771 (comment). Another advantage is that we don't have to hardcode any address or route (like slirp4netns would do), see also https://bugzilla.redhat.com/show_bug.cgi?id=2277954#c5. This is just the default: you can override the upstream interface with
Right, we realised just recently this isn't great for containers on trains or busses, see: #22737 (reply in thread) and following comments. I'm currently looking for a viable solution that doesn't break the whole model. The biggest problem I'm facing is that if we skip configuring addresses and routes because none were present on the host (for a given IP version), we risk making issues like #22197 worse: there, it's actually important that pasta refuses to wait until networking is ready (because, in that case, it will be ready, at some point). The proper solution for that issue would be in systemd (systemd/systemd#3312), but I'm not sure that will ever be addressed, so we can't plan on ignoring that, either. |
|
The address on the host can change during the lifetime of the container. If you want to avoid NAT and inherit the same IP on the container, then you're going to have to update the container's IP every time that the host IP changes. Perhaps it's feasible to assing non-routable IPv6 addresses (if those are the only available) and update the container with routable addresses when/if those are assigned on the host? In any case, using non-routable addresses would be better than using none, since currently the container is not reachable when using |
RIght, that was my idea to start with, but it comes with further complications, see #22737 (reply in thread).
That might be a good idea nevertheless, I'll need to check. Patches (tested, in this case ;)) are warmly welcome as well. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
Sorry I am not following the conversation here, is there actually a specific work item tracked here in either pasta or podman or can this be closed? |
|
@Luap99 Yes, this is still an issue. To summarise, if the host doesn't have a publicly routable IPv6 address when a container is started, the container cannot be reached from the host (with the default configuration). |
Kind of, in the sense that loosening start-up checks and admitting IPv6 addresses that are not routable is one of the bits that could improve support for the use case described at #22737 (reply in thread), in the short term.
...via IPv6, that is. |
This is what Firefox, curl any most other clients try by default. Note that the host is reachable, but refuses the connection, so there is never any reason for clients to retry using IPv4. |
Ouch, I missed this detail. |
I don't understand this part. A connection will always get connection refused when connecting to a local port where nothing is listening. So why should this ever be reason to not retry for curl, firefox, etc...? And trying this locally I see curl and firefox trying ::1 first and then fall back to 127.0.0.1, so what am I missing here? |
I looked into this, but if we include link-local addresses in the set "non-routable" ones we might accept to use (if we don't, that won't fix your use case), it might very well mean that we'll assign, or not, a global unicast address to the guest depending on timing (see also #22197). Should systemd/systemd#3312 ever get fixed, that would be much less critical and I would go ahead with this kind of change, but as long as it's not, it risks causing bigger issues. So I'd rather implement a more comprehensive fix that involves monitoring host addresses and routes via netlink. We started reporting some ideas and concerns in section 2. of this pad. |
|
I'm not sure that systemd/systemd#3312 would help. I suppose that you intend to monitor Regardless, such a solution would only work on configurations using systemd; the issue would still need a separate portable solution for other distributions.
This sounds a lot more reliable. |
|
I read all comments but I don't understand what's wrong with just binding to |
There is probably nothing wrong or inherently complicated about that, but pasta disables IPv6 altogether if it can't select an upstream interface with IPv6 support. This is just the simplest approach we went with, but indeed we could (probably should) define several levels of "IPv6 support", which is slightly more complicated because it raises questions such as: what do you advertise via NDP? Perhaps nothing? Should we answer neighbour solicitations at all? What do we map as source from the host in that case? A link-local address, I suppose? Feel free to send patches for that, by the way, if you have a tentative answer to those questions. :) |
|
Maybe it should do the same as slirp4netns does for local addresses, whatever that is. Starting my containers with slirp4netns, they just work. |
Except that slirp4netns doesn't support ipv6 port forwarding at all, we must use a extra port forwarder process to even have ipv6 support there (rootlessport). And that process is really more of a hack, it remaps ipv6 -> ipv4 in the container which well isn't right either (#14709) and it doesn't preserve the source ip of the original request which is what most user care about. And yes this stuff really should be documented (#22221) |
Same here with pasta, but my assumption was that if you use IPv6, your setup also has IPv6 connectivity, and this doesn't seem to be the case for many setups, yours included. No, we don't really have to reintroduce those buggy behaviours that @Luap99 just described. We can keep IPv6 enabled even if the setup has no global IPv6 connectivity, but we need to take care of a few details while doing that. |
|
It is interesting because I just started running containers on a clean Fedora 40. Only changed So it seems somehow resolved already? P.S. By default containers can't access the host. But reading the |
One thing we changed recently is that, while pasta decides if IPv6 support can be enabled, it now considers any host interface which has any route (not just default routes), see https://passt.top/passt/commit/netlink.c?id=450a6131beabd6537f2460bcc110a9a961697649. The package for which the version is relevant here is
Correct.
You need to use the address of the default gateway as seen from the container. You can't, of course, connect to localhost, because that's the container itself. The choice of using the default gateway to represent the host is arbitrary (see "Handling of traffic with local destination and source addresses" in pasta(1)) and we're working right now to make that configurable. |
So basically machine has a different setup and that's why it has IPv6 available. Thank you for the explanation! But still strange that the other machine does not get IPv6 enabled given it doesn't seem to have less ipv6 routes (update: ah, it uses an older I don't really understand the below also. How do I know default gateway of a container and can I access another container in the default network directly? btw I have stopped the
While from host: |
That's why: that is,
It depends on how you published the port.
Right, because 192.168.1.124 is this container itself (f64ea24695de).
Given your test below, this should work instead (see also a minimal example I posted at https://www.reddit.com/r/podman/comments/1c46q54/comment/kzppmpg/). But how did you publish the port for the second container? Did you bind it to a particular address?
|
|
The mariadb container was started with the command below. And from the host I can access it on any IPv4 or IPv6 host IP. |
This works for me: Then, as I resorted to Alpine Linux and telnet: I would suggest you take traffic captures on the loopback interface ( |
|
Thank you for the replies! Helped a lot. IPv6 appears to work without routable IPv6 IP. Now I observe something else strange with published ports. I don't know if it is a bug or an expected behavior. As a user thought, it is not very appreciated.
To start listener one could (remove network option for As for the client, I used netcat: Is this something that is expected to be fixed as part of this issue? Is there something I can do to make the ipv4 listener/ipv6 client combination work? -- this was performed with passt-0^20240821.g1d6142f-1.fc40.x86_64 |
Sorry, I missed this somehow. In general, trying to connect an IPv4 client to an IPv6 server, or the other way around, shouldn't work at all, but it (partially) works with loopback addresses (at least with pasta) because we just handle those as "loopback" without really caring about the exact address. I can reproduce the inconsistent behaviour you see, that's a separate issue I would say. Feel free to report it at bugs.passt.top (it's a pasta issue, you don't need Podman to reproduce it) or on this tracker. |
Cc: @dgibson |
For the moment being, we implemented a so-called local mode, which, if the network is not connected, assigns/uses link-local addresses for both IPv4 and IPv6. The containers will at least start and they will be addressable. This is not the full solution, yet, which will be based on the netlink monitor as I mentioned. |
Issue Description
Exposing a port via
--publishon a system with IPv6 doesn't work.The default network does not have IPv6 enabled by default, but I enabled it manually.
I edited
.local/share/containers/storage/networks/podman.jsonto include"ipv6_enabled": true,before starting the container to enable IPv6. Connections are now received by podman, but immediately dropped, and never reach the container.Steps to reproduce the issue
Steps to reproduce the issue
podman run --rm --publish 8001:8001 whynothugo/vdirsyncer-devkit-radicalecurl 'http://[::1]:8001'Describe the results you received
Describe the results you expected
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
No
Additional environment details
No response
Additional information
Attempting to use
localhostinstead of a specific IP fail too:I suppose that in some configurations
curlmight prefer IPv4 and it would work, but that's mostly luck.The text was updated successfully, but these errors were encountered: