Error Starting Control Plane w/ Podman on Fedora 33 #2213
Comments
|
you can try to use |
This results in the following:
I completely uninstalled and removed firewalld and installed iptables + iptables-service. Unfortunately, the problem still persists. |
you need the retain flag so the cluster is not deleted on failure |
|
Here's where things appear to go wrong and then a stack trace starts. |
|
v1/certificatesigningrequests": dial tcp [fc00:f853:ccd:e793::3]:6443: connect: connection refused Have a you done the iptables save working and nonworking that I mentioned previously? |
Where was that mentioned? Maybe I am stumbling over a similar issue on Manjaro Linux with podman 3.1.2. |
|
@aojea I can not get the control plane working and error messages in kindest/node journalctl look similar to the one mentioned here. Will try to provide full logs as described above. |
|
Just following up here, I still was unable to get podman to work. I had no issues with docker though. |
|
Same problem with Fedora 34. |
|
same here with Fedora 34. Fedora 32+ uses nftables per default instead of iptables, so maybe it is related this: #2271 (there is a PR pending, that adapts the iptables wrapper) |
|
I don't think that will change anything for podman because podman historically does not inject any iptables rules so we will not have anything to detect. That PR would change behavior an unlikely edge case where both type rules are in the node ns from the runtime which seems incredibly unlikely, but it's one of the few possible explanations for a different issue with rootless podman. For local podman we could maybe try to run some detection on the host as well but even then we also still need podman/docker + firewalld etc to be configured to work together with the same nft | iptables, which we've seen not be the case in the past on fedora. |
|
After ugrading podman recently I have run into problems myself. When using the v1.21.2 kind node image that contains the proper iptables wrapper script I appear to be reverting to having similar issues. $ podman run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume bf9ca67764a451fd098017594578ea5b2e35808047316865ed43bcbf89c649ba:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --publish=127.0.0.1:34813:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.21.2
ERRO[0002] error loading cached network config: network "kind" not found in CNI cache
WARN[0002] falling back to loading from existing plugins on disk
ERRO[0003] Error tearing down partially created network namespace for container 8779c1e9e30e664da84cc7d40537a104b44f99cffa07c44ed4354ce2d9644f34: error removing pod kind-control-plane_kind-control-plane from CNI network "kind": running [/usr/bin/iptables -t nat -D POSTROUTING -s 10.88.2.13 -j CNI-60d270dcc022372e968c855b -m comment --comment name: "kind" id: "8779c1e9e30e664da84cc7d40537a104b44f99cffa07c44ed4354ce2d9644f34" --wait]: exit status 2: iptables v1.8.7 (legacy): Couldn't load target `CNI-60d270dcc022372e968c855b':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
Error: error configuring network namespace for container 8779c1e9e30e664da84cc7d40537a104b44f99cffa07c44ed4354ce2d9644f34: error adding pod kind-control-plane_kind-control-plane to CNI network "kind": failed to list chains: running [/usr/bin/ip6tables -t nat -S --wait]: exit status 3: modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded. |
|
Above comment #2213 (comment) still applies, there's not a good way to reliably detect this without podman either explicitly providing an API to detect this or injecting a rule into the container. We'd probably accept (cc @aojea) PRs like:
But even that would only work at best for non-remote + rootful podman. Rootless and remote are still problematic. The podman driver could start an image (... which then becomes problematic as a dependency e.g. for offline ...) to probe the host network perhaps but rootless probably doesn't support this, and it would affect cluster creation time by adding another serial bottleneck before getting nodes booted. What would probably be most ideal is if |
|
Alternatively in the specific case that the legacy iptables modules literally do not exist, we can maybe try to probe for that case and assume the other mode should be used, that said we see real failures where the user e.g. requested an ipv6 cluster but their host doesn't have the ipv6 modules and it would probably be even more confusing if we switched to nftables, so maybe probe only ipv4 and only if we couldn't detect any rules from either. This also forces us to deviate from the kube-proxy logic, which would be slightly unfortunate, unless we can somehow convince them to add this rule there too (which ... it seems less clear that you'd ever need this in host network). |
|
I'm facing the same issue with a Fedora 34 virtual machine. Going through other issues, #2112 makes me wonder if this issue is related to btrfs. I'm not sure other Fedora users under this issue are using btrfs as it's default since Fedora 33. Out of curiosity, I made another vm with same Fedora 34 workstation installation and same 5.13.6 kernel version but choosing xfs manually instead of btrfs. And there is no issue on that. (Please refer to the attachment if you need log files from my kind export logs output) |
|
On my laptop running a brand new F34 install w/ btrfs, kind fails to start. On my workstation running F34 w/ xfs, kind starts fine. You might be onto something :) |
|
Willing to test any experimental changes and / or workarounds. Let me know what I can do to help. |
|
definitively last comments points to btrfs but that was fixed by ab52382 are you using latest kind version? I think we should lock this thread, this is becoming a placeholder of different errors and is very hard to follow |
|
Hey folks! 👋🏻 The Willing to help out with testing if needed! |
can you run that as root? rootless in podman has a known issue with a sidecar container used to configured the network, it was removed in master |
@aojea just making sure, when you say "run that as root" - you did mean simply with Thanks! |
|
weird, are you using fedora? |
|
Ha, that helped for this part even though I'm getting another error now. It seems like the container is failing to start for some reason |
|
I'm going to close this issue to avoid more confusion, no worries about that. Your error seems related to that #2445, AFAIK podman machine doesn't work |
|
@aojea: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Sorry for replying to closed issue: I'm just replying here as this is where google sent me when googling kind version 0.18.0 |
Thanks this also worked on my Arch based manjaro distribution: before that I always get this error: Then I follow the tip and it worked pretty well: |
What happened:
Hello, when trying to create a kind cluster via
kind create clusteron Fedora 33 w/ podman resulted in an error deploying the control plane.What you expected to happen:
I was hoping for a cluster to be created.
How to reproduce it (as minimally and precisely as possible):
kind create clusterAnything Else
Here are further log details
Environment:
kind version): kind v0.11.0-alpha go1.15.8 linux/amd64kubectl version): 1.21docker info):N/Apodman info):3.1.2/etc/os-release): Fedora 33The text was updated successfully, but these errors were encountered: