Podman pull behaviour different between podman pull and podman run

[penn5]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman run 10.0.2.2:5000/library/postgres:11 returns Error: initializing source docker://10.0.2.2:5000/library/postgres:11: pinging container registry 10.0.2.2:5000: Get "https://10.0.2.2:5000/v2/": http: server gave HTTP response to HTTPS client even though I've configured the source as insecure. Running the exact same command but changing run to pull works correctly.

Steps to reproduce the issue:

1. Setup an insecure registry that doesn't support HTTPS or has an insecure HTTPS configuration

2. Run an image in this registry via podman run

3. Fetch the image via podman pull

Describe the results you received:
The image is not fetched in the podman run command but subsequently downloads correctly in podman pull

Describe the results you expected:
The image download is the same whether implicit or explicit.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.4.1
API Version:  3.4.1
Go Version:   go1.16.8
Built:        Wed Oct 20 14:31:56 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.14-300.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1597345792
  memTotal: 2061529088
  ociRuntime:
    name: crun
    package: crun-1.2-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.2
      commit: 4f6c8e0583c679bfee6a899c05ac6b916022561b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 11m 42.16s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  10.0.2.2:5000:
    Blocked: false
    Insecure: true
    Location: 10.0.2.2:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: 10.0.2.2:5000
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.1
  Built: 1634740316
  BuiltTime: Wed Oct 20 14:31:56 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3:3.4.1-1.fc34.x86_64
podman-plugins-3:3.4.1-1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes, adding --tls-verify=false works around the issue

Additional environment details (AWS, VirtualBox, physical, etc.):
QEMU Fedora CoreOS

[vrothberg]

Thanks for reaching out. The issue has already been fixed in the main branch via #12064.

@mheon I don't see a backport to v3.4. Should we prepare one?

[mheon]

If you're willing to do the c/common backports, sure.

[vrothberg]

If you're willing to do the c/common backports, sure.

containers/common#840

[vrothberg]

Here's the backport to v3.4 -> #12451