podman machine - can't run pods

[christian-korneck]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I'm running into the following issue using podman v3.3.1, installed via homebrew, on macOS 11.5.2 x86_64.

I've initialized/started a podman machine VM with podman machine init && podman machine start (which is running and I can run podman ps from the host and I can podman ssh into it and it appears the VM has network connectivity). I now expect to be able to run podman run hello-world, but it fails:

$ podman run --rm -ti hello-world --log-level=debug
Error: failed to parse "X-Registry-Auth" header for /v3.3.1/libpod/images/pull?alltags=false&arch=&authfile=&os=&password=&policy=missing&quiet=false&reference=hello-world&username=&variant=: error storing credentials in temporary auth file (server: "https://index.docker.io/v1/", user: ""): key https://index.docker.io/v1/ contains http[s]:// prefix

(the same error happens when trying to just run a podman pull hello-world)

When I try the same with the podman cli inside the VM the image pull works and running the VM fails with a CNI related error:

$ podman machine ssh
$$ podman pull hello-world
Trying to pull docker.io/library/hello-world:latest...
[...]
d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726

$$ podman run --rm -ti hello-world
ERRO[0000] error loading cached network config: network "podman" not found in CNI cache
WARN[0000] falling back to loading from existing plugins on disk
Error: error configuring network namespace for container ae1d39202992c010a99ba5e1e1d36e9ceaf097ccc149475c1448a606d0990748: error adding pod elated_bassi_elated_bassi to CNI network "podman": unexpected end of JSON input

Describe the results you received:
running podman run hello-world with a podman machine VM failed

Describe the results you expected:
running podman run hello-world with a podman machine VM works

Additional information you deem important (e.g. issue happens only occasionally):

the net podman exists:

$$ podman network ls
NETWORK ID    NAME        VERSION     PLUGINS
2f259bab93aa  podman      0.4.0       bridge,podman-machine,portmap,firewall,tuning

net details:

$ podman network inspect podman
[
    {
        "cniVersion": "0.4.0",
        "name": "podman",
        "plugins": [
            {
                "bridge": "cni-podman0",
                "hairpinMode": true,
                "ipMasq": true,
                "ipam": {
                    "ranges": [
                        [
                            {
                                "gateway": "10.88.0.1",
                                "subnet": "10.88.0.0/16"
                            }
                        ]
                    ],
                    "routes": [
                        {
                            "dst": "0.0.0.0/0"
                        }
                    ],
                    "type": "host-local"
                },
                "isGateway": true,
                "type": "bridge"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "podman-machine"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "portmap"
            },
            {
                "type": "firewall"
            },
            {
                "type": "tuning"
            }
        ]
    }
]

Output of podman version:

on the macOS host:

$ podman version
Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Mon Aug 30 21:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.0
API Version:  3.3.0
Go Version:   go1.16.6
Built:        Fri Aug 20 21:36:14 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.12-200.fc34.x86_64
  linkmode: dynamic
  memFree: 1599819776
  memTotal: 2061852672
  ociRuntime:
    name: crun
    package: crun-0.21-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.21
      commit: c4c3cdf2ce408ed44a9e027c618473e6485c635b
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 25m 28.41s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.0
  Built: 1629488174
  BuiltTime: Fri Aug 20 19:36:14 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.0

Package info (e.g. output of rpm -q podman or apt list podman):

$ brew list podman
/usr/local/Cellar/podman/3.3.1/bin/gvproxy
/usr/local/Cellar/podman/3.3.1/bin/podman
/usr/local/Cellar/podman/3.3.1/bin/podman-remote
/usr/local/Cellar/podman/3.3.1/etc/bash_completion.d/podman
/usr/local/Cellar/podman/3.3.1/share/fish/vendor_completions.d/podman.fish
/usr/local/Cellar/podman/3.3.1/share/man/ (159 files)
/usr/local/Cellar/podman/3.3.1/share/zsh/site-functions/_podman

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

This isn't a vanilla macOS machine, but I've removed the following paths prior the above test run:

• ~/.ssh/podman*
• ~/.config/containers
• ~/.local/share/containers

[afbjorklund]

I had the same issue, and it was caused by having a ~/.docker/config.json file.

Apparently Podman tries to read the client config from Docker, but fails to use it ?

It could be the credential store that is confusing it, or just "reading other people's mail"

{
	"auths": {
		"https://index.docker.io/v1/": {}
	},
	"credsStore": "secretservice",
	"experimental": "enabled"
}

(removed most of the irrelevant config)

https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Moving the config file to the side made the podman-remote commands work again.

[christian-korneck]

@afbjorklund thanks a lot, removing ~/.docker/config.json fixes the issue. (As this seems unintended behavior , I guess /kind bug is still appropriate?)

[afbjorklund]

Seems to be a duplicate of #11235

[christian-korneck]

Seems to be a duplicate of #11235

Thanks again. /close