Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

podman run --rm -it alpine sh (apk update or any egress doesn't work) #7613

Closed
bobhenkel opened this issue Sep 11, 2020 · 15 comments
Closed
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@bobhenkel
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
I'm running podman run --rm -it debian sh and podman run --rm -it alpine sh with the root account on a ubuntu 20.04 instance. When running alpine and debian images I can't resolve addresses from inside the container, but with the ubuntu image I can. From the host OS I'm able to hit the internet.

From host OS:

ubuntu@test-ubunt20:~$ wget http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
--2020-09-11 21:39:43--  http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
Resolving dl-cdn.alpinelinux.org (dl-cdn.alpinelinux.org)... 199.232.6.133, 2a04:4e42:3e::645
Connecting to dl-cdn.alpinelinux.org (dl-cdn.alpinelinux.org)|199.232.6.133|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 634425 (620K) [application/octet-stream]
Saving to: ‘APKINDEX.tar.gz’

APKINDEX.tar.gz                             100%[========================================================================================>] 619.56K  1.06MB/s    in 0.6s    

Steps to reproduce the issue:

  1. ssh into ubuntu 20.04 host as ubuntu user
  2. sudo su or just as ubuntu user(both will result in same out come)
  3. podman run --rm -it alpine sh
  4. apk update

Describe the results you received:

root@test-ubunt20:/home/ubuntu# podman run --rm -it alpine sh
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.12/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.2c4ac24e.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.12/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.40a3604f.tar.gz: No such file or directory
2 errors; 14 distinct packages available
/ # ^C

Describe the results you expected:
I was expecting for apk upgrade command to be able to reach the internet and update the packages. The odd thing is
if I do the same with the ubuntu image apt-get update works fine. However if I try with the debian image apt update also fails as it can't resolve.

root@test-ubunt20:/home/ubuntu# podman run --rm -it debian sh
Trying to pull docker.io/library/debian...
Getting image source signatures
Copying blob 57df1a1f1ad8 done  
Copying config f6dcff9b59 done  
Writing manifest to image destination
Storing signatures
# apt update
Err:1 http://security.debian.org/debian-security buster/updates InRelease
  Temporary failure resolving 'security.debian.org'
Err:2 http://deb.debian.org/debian buster InRelease                  
  Temporary failure resolving 'deb.debian.org'

Additional information you deem important (e.g. issue happens only occasionally):
Happens 100% of the time.

Output of podman version:

root@test-ubunt20:/home/ubuntu# podman version
Version:      2.0.6
API Version:  1
Go Version:   go1.14.2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

root@test-ubunt20:/home/ubuntu# podman info --debug
host:
  arch: amd64
  buildahVersion: 1.15.1
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 8
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: test-ubunt20
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.0-37-generic
  linkmode: dynamic
  memFree: 11541790720
  memTotal: 12562845696
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 3e46dd849fdf6bfa68127786e073318184641f05
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 3h 9m 13.47s (Approximately 0.12 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.2
  OsArch: linux/amd64
  Version: 2.0.6

Package info (e.g. output of rpm -q podman or apt list podman):

root@test-ubunt20:/home/ubuntu# apt list podman
Listing... Done
podman/unknown,now 2.0.6~2 amd64 [installed]
podman/unknown 2.0.6~2 arm64
podman/unknown 2.0.6~2 armhf
podman/unknown 2.0.6~2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes/Yes

Additional environment details (OpenStack):

@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Sep 11, 2020
@Luap99
Copy link
Member

Luap99 commented Sep 11, 2020

What is the output of cat /etc/resolv.conf inside the container ?

@bobhenkel
Copy link
Author

bobhenkel commented Sep 11, 2020

ubuntu@test-ubunt20:~$ podman run --rm -it alpine sh
/ # cat /etc/resolv.conf
search openstacklocal us.kuberchaun.com americas.kuberchaun.com kuberchaun.com dev.kuberchaun.com
nameserver 10.0.2.3
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844
options edns0
/ # exit

And from ubuntu image where things appear to work.

ubuntu@test-ubunt20:~$ podman run --rm -it ubuntu sh
Trying to pull docker.io/library/ubuntu...
Getting image source signatures
Copying blob 54ee1f796a1e done  
Copying blob f7bfea53ad12 done  
Copying blob 46d371e02073 done  
Copying blob b66c17bbf772 done  
Copying config 4e2eef94cd done  
Writing manifest to image destination
Storing signatures
# cat /etc/resolv.conf
search openstacklocal us.kuberchaun.com americas.kuberchaun.com kuberchaun.com dev.kuberchaun.com
nameserver 10.0.2.3
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844
options edns0
#

@zhangguanzhang
Copy link
Collaborator

could you try this:

podman run --rm -it --net host alpine sh
apk update

@bobhenkel
Copy link
Author

That seems to work. Does that shed some light on the underlying reason for the issue? No issue with ubuntu image, without the --net host.

ubuntu@test-ubunt20:~$ podman run --rm -it --net host alpine sh
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
v3.12.0-313-g23f4082141 [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
v3.12.0-310-gfeadd8fa34 [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
OK: 12746 distinct packages available
/ #

@bobhenkel
Copy link
Author

bobhenkel commented Sep 13, 2020

So I read issue #3277 and a comment suggested adding --dns 10.0.2.3 to the podman run command to force the dns being used, which allowed apk update to pull down the payload from the internet. It seems on my ubuntu host /etc/resolv.conf has nameserver 127.0.0.53 where as my sles15sp2 host which doesn't show this issue doesn't have the dns set to local(127.0.0.53). So unless I uncovered something of value for the podman devs, I think this issue can be closed. Though it still feels a little odd, for someone coming from docker I haven't experienced this issue. Though maybe I should re-test that statement and see if my ubuntu host with docker installed has the same issue.

@mheon
Copy link
Member

mheon commented Sep 13, 2020 via email

@bobhenkel
Copy link
Author

"Are the contents of resolve.conf different based on the image you run?"
No.
So what allows the ubuntu image to hit the internet while the alpine not to? I mis-stated earlier in the thread that this also happens on the debian image, I just ran it and was able to to apt update just fine and was able to yum update on a fedora image just fine as well.

ubuntu@test-ubunt20:~$ podman run --rm -it alpine sh
/ # cat /etc/resolv.conf 
search openstacklocal us.kuberchaun.com americas.kuberchaun.com kuberchaun.com dev.kuberchaun.com
nameserver 10.0.2.3
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844
options edns0
/ # exit
ubuntu@test-ubunt20:~$ podman run --rm -it ubuntu bash
root@f72fd4750df8:/# cat /etc/resolv.conf 
search openstacklocal us.kuberchaun.com americas.kuberchaun.com kuberchaun.com dev.kuberchaun.com
nameserver 10.0.2.3
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844
options edns0
root@f72fd4750df8:/#

@zhangguanzhang
Copy link
Collaborator

could you try this:

podman run --rm -ti  docker.io/frolvlad/alpine-glibc sh
apk update

@bobhenkel
Copy link
Author

/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.12/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.2c4ac24e.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.12/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.40a3604f.tar.gz: No such file or directory
2 errors; 17 distinct packages available
/ # 

@dR3b
Copy link

dR3b commented Sep 28, 2020

From my Arch Linux:

~ » podman run --rm -ti  docker.io/frolvlad/alpine-glibc sh
/ # cat /etc/resolv.conf
nameserver 10.0.2.3
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844
/ #
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.12/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.2c4ac24e.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.12/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.40a3604f.tar.gz: No such file or directory
2 errors; 17 distinct packages available
~ » podman run --rm -it --net=host alpine sh
/ # cat /etc/resolv.conf
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.12/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.2c4ac24e.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.12/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.40a3604f.tar.gz: No such file or directory
2 errors; 14 distinct packages available
~ » podman info
host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 35a2fa83022e56e18af7e6a865ba5d7165fa2a4a'
  cpus: 8
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: user1-nb1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.12-arch1-1
  linkmode: dynamic
  memFree: 12956086272
  memTotal: 16428888064
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 29m 56.03s
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /home/user1/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/user1/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  volumePath: /home/user1/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601135437
  BuiltTime: Sat Sep 26 17:50:37 2020
  GitCommit: 9f6d6ba0b314d86521b66183c9ce48eaa2da1de2
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

@dR3b
Copy link

dR3b commented Sep 29, 2020

It works now (systemd-resolved.service | /etc/resolv.conf):

nameserver 127.0.0.53
options edns0 trust-ad

@rhatdan
Copy link
Member

rhatdan commented Sep 29, 2020

@dR3b @bobhenkel Can I close this issue then?

@dR3b
Copy link

dR3b commented Sep 29, 2020

@rhatdan
Sure thanks!

@zhangguanzhang
Copy link
Collaborator

/close

@openshift-ci-robot
Copy link
Collaborator

@zhangguanzhang: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests

7 participants