unable to detect hairpin mode - iptables >1.8.4 update issues #3940
Comments
|
Thank you! |
dragoangel
changed the title
|
Issue also reproduce on Debian 11 |
|
As discussed in telegram group, here's the workaround for those using Debian 11 with iptables version > 1.8.4. Check your version by typing: Download older packages and dependencies of iptables (in this example my machine usues architecture amd64): After downgrading, the error messages disappeared and IPv6 mail transmission started working again. Big thanks to @dragoangel for your help! |
|
On Oracle Linux 8 you can just do "yum downgrade iptables" and it will revert back to iptables-1.8.4-15.0.1.el8.x86_64 which is not affected. Not sure if the same works for Centos |
|
Hi @mcmufffin, for CentOS8 I described all flow in details ;) |
Hehe, yeah i saw that, but downloading rpm by hand? Ill give that a pass if it works via yum downgrade :) |
|
Downgrade not guaranty it will be downgraded to needed version ;) |
|
1.8.4-15.0.1 seem also not to be affacted. |
|
According to robbertkl/docker-ipv6nat#67, a patch is being backported for CentOS... Does anyone know the state of a patch for Debian Bullseye (11)? |
You got that mixed up, the problem was not solved by that mentioned patched but rather indstroduced by it, so CentOS is still not fixed. It seems like Iptables 1.8.7 includes a fix so it should work on debian 11 bullseye. |
|
Oh! Oops.
I still got the "unable to detect hairpin mode" message on Debian 11 with the latest packages including iptables 1.8.7 😢 Downgrading to the 1.8.2 packages mentioned earlier in the thread 'fixed' it. |
|
Also, for now I've disabled the Docker userland proxy. Having no IPv6 connectivity is better than having broken IPv6 connectivity with an open relay. |
|
To add: I now see in nginx logs internal mailcow IPv6 and non public one, logically postfix (and dovecot, etc) should also see same and result in open relay, but postfix (and other containers) see IPv6 correctly. Really strange... |
|
Ok, found out where the dog is buried:
Adding them also failed: and in result: If somebody have ideas how fix it, please advice. In general need move somewhere else :/ |
|
On my system there are ipv6 rules missing too. I've force updated mailcow and downgraded iptables. Everything seems to work, but the web ports 80 and 443 are missing so that the web ui isn't available over ipv6. ip6tables -L -vn |
|
ya, still getting this on debian bullseye, tried setting alternatives to legacy for [ip,ip6,eb]tables a la |
@prometheanfire You'll have to downgrade iptables and related packages. See @henrocker's comment from February 14. |
|
ah, missed that, it works, it's dirty, but it works... |
|
@prometheanfire in start I doesn't know Debian 11 is not stable in all terms - it not yet release. You mustn't use it anywhere except testing. Debian 11 not have docker support, so it mean you mustn't run mailcow on it in short. |
|
I'm aware that it's not stable. Using docker from the base debian repo works. I'm not running mailcow (using ipv6-nat for prometheus blackbox exporter). That said, I'll be migrating mailcow 'soon' as well :P |
|
@dragoangel For what it's worth, I've been using Debian "testing" in production for maybe 15 years now. I haven't experienced any major issues (given the packages need to sit in "unstable" for 10 days with no high-pri bugs before migrating to testing) and the packages are usually a lot newer than stable. Docker runs fine on Debian 11. There's a warning that cgroup 2 support is experimental but it doesn't seem to break anything. |
|
There isn't even an official Docker package.
If you are using testing in production that's what you should expect.
… Am 30.03.2021 um 06:59 schrieb Daniel Lo Nigro ***@***.***>:
@dragoangel For what it's worth, I've been using Debian "testing" in production for maybe 15 years now. I haven't experienced any major issues. Docker runs fine on Debian 11. There's a warning that cgroup 2 support is experimental but it doesn't seem to break anything.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
@andryyy The Buster Docker packages work fine on Bullseye too. If they were incompatible (eg. incompatible versions of dependencies like libc), you'd get an error when installing the packages, but they install and run with no problems. |
|
You should never do that. Just because it installs doesn't mean it works. Really. Never do that. Bullseye and Buster are pretty different from each other. Especially when it comes to networking and containers. |
|
@andryyy Normally I'd agree, however bullseye is getting closer to release (they've already commenced the soft freeze) so more and more software is bullseye-compatible now. Given Docker runs on other bleeding-edge Linux distros like Arch and Fedora, I'm not aware of any major incompatibilities around networking subsystems in newer distros, other than this issue with iptables + IPv6 (which is for ipv6nat which isn't even an official Docker feature). I do agree that in most cases it's better to use stable, but using testing is usually fine too. |
|
There is a reason there is no bullseye package yet. Same applied to CentOS 8. C7 packages installed on C8 but had many, many flaws. You cannot say "ha, well, it installs fine, so all dependencies are okay, should work!". That's just wrong. Just because it installs doesn't mean it's fine. Never. And I recommend to NEVER run testing in prod and even worse, NEVER install unsupported packages from other releases. All packages differ a lot. It's not that Docker is just a binary/a daemon that runs or doesn't run. Much more is happening in the background. |
|
Sure, that makes sense. What about using the Docker package bundled with Debian? Docker 20.10.3 is included in Bullseye: https://packages.debian.org/bullseye/docker.io. There should not be any incompatibilities in that case, and in general I trust Debian's packages more than vendor packages (since I know they're following DFSG, their builds are repeatable, etc)
It's no riskier than running Arch in prod, which a lot of people do :). It's not as stable as Debian stable, but it's about on par with Arch and Fedora. |
|
It's still testing... I mean... see: It's working in buster. It is not working in bullseye. Bullseye is testing. So we should probably wait? Check https://download.docker.com/linux/debian/dists/ for updates. :) Do not use the pre-packaged Docker daemon, please. Same applies to Rspamd. They offer Rspamd in their repository which is totally broken and outdated. |
|
@andryyy Also note that the 'broken' version of iptables is present in buster-backports (https://packages.debian.org/buster-backports/iptables) meaning the original issue is possibly still an issue even on some Buster systems that are using backported packages.
Why? The Debian team work hard to maintain the packages provided by the Debian project. Is it just because the version in stable is old?
Packages in Debian stable being outdated is 'intentional' in that they stick to older versions (LTS releases if the upstream provides them). That's why people run Debian 'buster-backports' or 'testing' (Bullseye) in prod - To get newer packages that may not be as stable as the packages in 'stable'. Essentially you choose between stability (stable) and cutting-edge features (testing or even unstable) |
|
@Daniel15 What exactly are you trying to achieve with the hole "stable <--> unstable" dicussion? Its your system, you can run whatever you want there, but if you do run version mailcow is not tested against, you might run into mentioned bugs, so dont expect support for that. |
|
@mcmufffin It just started from a comment someone posted earlier saying to not use Debian 11 in production. Perhaps it's a bit off-topic for this ticket, but Debian 11 is one of the environments that is experiencing this bug. If nobody was running bleeding-edge OSes in production, this bug wouldn't be an issue 😄
Isn't one of the main reasons for using Docker to provide a layer of abstraction between the underlying OS and the app? Perhaps Mailcow is more sophisticated than other Docker images though, given its usage of iptables (eg for fail2ban). |
|
@Daniel15, ipv6nat is hardly rely on iptables as it need to set nat & snat, and here is the risk: with broken iptables rules masquerade can be not applied and you will get open relay on 25 port, this is the main risk of unsupported OS. Fail2ban here isn't a problem. Most docker about which you think/speak don't care much from which ip client connecting and from what ip docker need to do outgoing connections. This not the case for mail server ;) so please let's stop. P.s.: this not mailcow specific issue, that why on this issue upstream tag is placed, issue has just informational purpose till it will be not resolved. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
github-actions
bot
added
the
stale
|
/me really doesn't know why this bot exists |
andryyy
removed
the
stale
|
Just to note issue still persists with |
|
Debian Bullseye is going to be frozen on 2021-07-17, and the release is planned for the end of July. I hope this issue is fixed by then, as otherwise a lot of people will start encountering it as they upgrade their systems 😞 |
|
People should use it when there are stable, official Docker packages.
… Am 17.06.2021 um 01:48 schrieb Daniel Lo Nigro ***@***.***>:
Debian Bullseye is going to be frozen on 2021-07-17, and the release is planned for the end of July. I hope this issue is fixed by then, as otherwise a lot of people will start encountering it as they upgrade their systems 😞
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
@andryyy Docker has already been officially updated to support Bullseye, in preparation for the release. See https://docs.docker.com/engine/install/debian/ So installing Docker's official packages on Bullseye is supported today, even though Bullseye is still a "testing" build (although it does have a release candidate now). |
But Debian bullseye is NOT supported (until it's released) and so it's not recommended to run in production. And only the installer has a RC, not the whole distribution. |
|
@waja Sure, I don't disagree with you :) I just meant that Docker can run on Bullseye, and it'll be a fully supported stack (both the OS and Docker) the day that Bullseye is released as stable, since the Docker packages for Bullseye already exist. If this incompatibility with iptables 1.8.4+ is still around once Bullseye is released, a lot of people are going to encounter it. Downgrading the package is a lot sketcher then, as you'd need to pull the package from oldstable which isn't always going to work. |
|
Shouldn't this have been fixed with iptables 1.8.7, which is the version included in bullseye? @Daniel15, if it's not fixed, please file a bug report against the package immediately. We certainly don't want bullseye to get released like this. The only thing Debian could do is backport a patch from a newer iptables version, but for that you would need to identify which post-1.8.7 version fixes the problem. |
|
Ah, the problem might be that ipv6nat is built with Alpine 3.12 (and our netfilter container with 3.13, by the way), which includes iptables 1.8.6. So all we will need to do is upgrade to Alpine 3.14 and it should be fine. No bug in Debian 11. |
|
@mkuron I plan to make Docker > 20.x (not sure which minor) a requirement, kick ipv6nat and auto-config the dockerd. I think that's the correct step forward. |
Sounds reasonable in the long run, but it‘s too early. IPv6 NAT in Docker is still an experimental feature and was severely broken in the early versions. Not sure how broken it is now, but we wouldn‘t want to require use of an experimental feature.
I‘m not a big fan of automatically modifying system-level config files. Perhaps just stick to checking whether it has been configured and throwing an error message if not. Or better, file a bug with Docker that this should be configurable per-container and then file a bug with docker-compose that this should be part of the compose file. |
|
@Daniel15 or other guys, can you please check @geektoor patched image on Debian? For me in centos8 it resolve issues.
Post here result: issue resolved or persists. Will be cool if you reply in PR mentioned above about your testing result as well 😄 |
|
Well PR megred 🙌 , so soon there will be image from robbertkl 😄 |
|
Resolved by upstream |
New iptables-1.8.4-15.el8_3.3 packages on CentOS8 leading to non working ipv6nat docker.
mailcow users on CentOS8 who have IPv6 enabled need to lock iptables to latest working version till issue will not be resolved.
Issue described here: robbertkl/docker-ipv6nat#67
If you not yet updated to 1.8.4.-15.el8_3.3 please lock packages version:
sudo dnf install -y 'dnf-command(versionlock)'; sudo dnf versionlock add iptables iptables-ebtables iptables-libs iptables-servicesIf you already updated iptables you need to downgrade packages and reboot the server:
docker-compose downin mailcow home directorysudo dnf install -y 'dnf-command(versionlock)'; sudo dnf versionlock add iptables iptables-ebtables iptables-libs iptables-servicesdocker-compose up -din mailcow home directoryTo unlock iptables packages when issue will be resolved:
sudo dnf versionlock del iptables iptables-ebtables iptables-libs iptables-servicesNote: some users can not have
iptables-servicespackage, this totally fine, downgrade command will not touch it in this case and downgrade all other packages iptables packages successfully.The text was updated successfully, but these errors were encountered: