Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

podman 4.0 hangs indefinitely if ipv6 is disabled on system #13388

Closed
craftyguy opened this issue Mar 1, 2022 · 13 comments · Fixed by #13485
Closed

podman 4.0 hangs indefinitely if ipv6 is disabled on system #13388

craftyguy opened this issue Mar 1, 2022 · 13 comments · Fixed by #13485
Assignees
Labels
In Progress This issue is actively being worked by the assignee, please do not work on this at this time. kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@craftyguy
Copy link
Contributor

craftyguy commented Mar 1, 2022

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Podman 4.0.1 will hang indefinitely with this warning if ipv6 is disabled on the system:

level=warning msg="failed to set net.ipv6.conf.default.accept_dad sysctl: open /proc/sys/net/ipv6/conf/default/accept_dad: no such file or directory"

Steps to reproduce the issue:

  1. disable ipv6 (e.g. with kernel parameter ipv6.disable=1

  2. podman run --rm hello-world

Describe the results you received:

podman doesn't run the simple hello-world container

Describe the results you expected:

podman to run the container. I expect that warnings don't hang the application, anything that is required and missing should be an error, not a warning. I also expect podman to actually work without ipv6 enabled :)

Additional information you deem important (e.g. issue happens only occasionally):

happens every time on an Alpine Linux host. podman 3.x did not have this issue

Output of podman version:

Client:       Podman Engine
Version:      4.0.1
API Version:  4.0.1
Go Version:   go1.17.7
Git Commit:   c0e1dde8334e8a3a2a2bb025075d8ed68574ee33
Built:        Mon Feb 28 18:03:49 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

❯ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.0-r0
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: feb71f1a6023ee6d874ed0a62a46a205a92f5dfc'
  cpus: 12
  distribution:
    distribution: alpine
    version: 3.15.0
  eventLogger: file
  hostname: librem14
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.24-2-lts
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 30535794688
  memTotal: 33538830336
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.2-r0
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /tmp/1000-runtime-dir/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-r0
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 8384409600
  swapTotal: 8384409600
  uptime: 4m 52.37s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/clayton/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/clayton/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/1000-runtime-dir/containers
  volumePath: /home/clayton/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 1646100229
  BuiltTime: Mon Feb 28 18:03:49 2022
  GitCommit: c0e1dde8334e8a3a2a2bb025075d8ed68574ee33
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

❯ apk info podman
podman-4.0.1-r1 description:
Simple management tool for pods, containers and images

podman-4.0.1-r1 webpage:
https://podman.io/

podman-4.0.1-r1 installed size:
49 MiB

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Alpine Linux Edge, x86_64, linux-lts kernel @ 5.15.24

strace output: strace.log

@openshift-ci openshift-ci bot added kind/bug Categorizes issue or PR as related to a bug. kind/feature Categorizes issue or PR as related to a new feature. labels Mar 1, 2022
@mheon mheon removed the kind/feature Categorizes issue or PR as related to a new feature. label Mar 1, 2022
@mheon
Copy link
Member

mheon commented Mar 1, 2022

@baude @Luap99 PTAL

@Luap99 Luap99 self-assigned this Mar 1, 2022
@jewthon
Copy link

jewthon commented Mar 6, 2022

Plz assign me this issue . I want to work on it @Luap99 @baude

@Luap99 Luap99 assigned craftyguy and jewthon and unassigned Luap99 and craftyguy Mar 6, 2022
@Luap99
Copy link
Member

Luap99 commented Mar 6, 2022

@jewthon You got it, feel free to ping me if you need help.

@rhatdan
Copy link
Member

rhatdan commented Mar 7, 2022

We need this pretty quick though, I have a feeling a lot of users do this. Especially in security consious locations where they don't use IPV6

@baude
Copy link
Member

baude commented Mar 7, 2022

shouldn't this be implemented in netavark ?

@Luap99
Copy link
Member

Luap99 commented Mar 7, 2022

no this is a slirp4netns ipv6 hack

@Luap99
Copy link
Member

Luap99 commented Mar 8, 2022

the workaround is to set network_cmd_options = [] under the [engine] section in ~/.config/containers/containers.conf

@jewthon
Copy link

jewthon commented Mar 10, 2022 via email

@Luap99
Copy link
Member

Luap99 commented Mar 10, 2022

You should ignore ENOENT here:


But the more important part is to fix the hang, the way how slirpReadyChan is completely broken. It also does not actually check that the sysctl is set before we start slirp4netns. So I think fixing the synchronization is more complicated.

@baude
Copy link
Member

baude commented Mar 11, 2022

user has reported this again on irc; any progress?

@jewthon
Copy link

jewthon commented Mar 11, 2022 via email

@Luap99
Copy link
Member

Luap99 commented Mar 11, 2022

I will fix it.

@Luap99 Luap99 assigned Luap99 and unassigned jewthon Mar 11, 2022
@Luap99 Luap99 added the In Progress This issue is actively being worked by the assignee, please do not work on this at this time. label Mar 11, 2022
@Luap99
Copy link
Member

Luap99 commented Mar 11, 2022

#13485 should fix it. It would be great if someone could test this on a system with ipv6 disabled.

Luap99 added a commit to Luap99/libpod that referenced this issue Mar 14, 2022
When enable_ipv6=true is set for slirp4netns (default since podman v4),
we will try to set the accept sysctl. This sysctl will not exist on
systems that have ipv6 disabled. In this case we should not error and
just ignore the extra ipv6 setup.

Also the current logic to wait for the slirp4 setup was kinda broken, it
did not actually wait until the sysctl was set before starting slirp.
This should now be fixed by using two `sync.WaitGroup`s.

[NO NEW TESTS NEEDED]

Fixes containers#13388

Signed-off-by: Paul Holzinger <[email protected]>
tricktron pushed a commit to tricktron/podman that referenced this issue Mar 14, 2022
When enable_ipv6=true is set for slirp4netns (default since podman v4),
we will try to set the accept sysctl. This sysctl will not exist on
systems that have ipv6 disabled. In this case we should not error and
just ignore the extra ipv6 setup.

Also the current logic to wait for the slirp4 setup was kinda broken, it
did not actually wait until the sysctl was set before starting slirp.
This should now be fixed by using two `sync.WaitGroup`s.

[NO NEW TESTS NEEDED]

Fixes containers#13388

Signed-off-by: Paul Holzinger <[email protected]>
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/podman that referenced this issue Mar 17, 2022
When enable_ipv6=true is set for slirp4netns (default since podman v4),
we will try to set the accept sysctl. This sysctl will not exist on
systems that have ipv6 disabled. In this case we should not error and
just ignore the extra ipv6 setup.

Also the current logic to wait for the slirp4 setup was kinda broken, it
did not actually wait until the sysctl was set before starting slirp.
This should now be fixed by using two `sync.WaitGroup`s.

[NO NEW TESTS NEEDED]

Fixes containers#13388

Signed-off-by: Paul Holzinger <[email protected]>
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/podman that referenced this issue Mar 18, 2022
When enable_ipv6=true is set for slirp4netns (default since podman v4),
we will try to set the accept sysctl. This sysctl will not exist on
systems that have ipv6 disabled. In this case we should not error and
just ignore the extra ipv6 setup.

Also the current logic to wait for the slirp4 setup was kinda broken, it
did not actually wait until the sysctl was set before starting slirp.
This should now be fixed by using two `sync.WaitGroup`s.

[NO NEW TESTS NEEDED]

Fixes containers#13388

Signed-off-by: Paul Holzinger <[email protected]>
mheon pushed a commit to mheon/libpod that referenced this issue Mar 30, 2022
When enable_ipv6=true is set for slirp4netns (default since podman v4),
we will try to set the accept sysctl. This sysctl will not exist on
systems that have ipv6 disabled. In this case we should not error and
just ignore the extra ipv6 setup.

Also the current logic to wait for the slirp4 setup was kinda broken, it
did not actually wait until the sysctl was set before starting slirp.
This should now be fixed by using two `sync.WaitGroup`s.

[NO NEW TESTS NEEDED]

Fixes containers#13388

Signed-off-by: Paul Holzinger <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
In Progress This issue is actively being worked by the assignee, please do not work on this at this time. kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants