Error initializing source docker://localhost/myapp:latest

[djnotes]

I generated a pod description using podman generate kube mypod > mypod.yaml from two containers (one local image and the other is mariadb). Now I want to deploy the pod using this description by using podman play kube mypod.yaml, but I get the following error:

Trying to pull localhost/myapp:latest...
Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
Error: unable to pull localhost/myapp:latest: Error initializing source docker://localhost/myapp:latest: error pinging docker registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused

Here's the content of mypod.yaml:

# Generation of Kubernetes YAML is still under development!
#
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-2.0.6
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2020-09-19T14:47:34Z"
  labels:
    app: myapp-pod
  name: myapp-pod
spec:
  containers:
  - command:
    - python3
    - app.py
    env:
    - name: PATH
      value: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    - name: TERM
      value: xterm
    - name: GPG_KEY
      value: E3FF2839C048B25C084DEBE9B26995E310250568
    - name: PYTHON_VERSION
      value: 3.8.5
    - name: PYTHON_GET_PIP_URL
      value: https://github.com/pypa/get-pip/raw/5578af97f8b2b466f4cdbebe18a3ba2d48ad1434/get-pip.py
    - name: PYTHON_GET_PIP_SHA256
      value: d4d62a0850fe0c2e6325b2cc20d818c580563de5a2038f917e3cb0e25280b4d1
    - name: LANG
      value: C.UTF-8
    - name: PYTHON_PIP_VERSION
      value: 20.2.2
    - name: container
      value: podman
    - name: HOSTNAME
      value: myapp-pod
    image: localhost/myapp:latest
    name: myapp-app
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
      seLinuxOptions: {}
    workingDir: /app
  - command:
    - mysqld
    env:
    - name: PATH
      value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    - name: TERM
      value: xterm
    - name: MARIADB_VERSION
      value: 1:10.5.5+maria~focal
    - name: MYSQL_USER
      value: myapp
    - name: MYSQL_PASSWORD
      value: p@ssWord
    - name: MYSQL_ROOT_PASSWORD
      value: p@ssWord
    - name: MYSQL_DATABASE
      value: myapp
    - name: GPG_KEYS
      value: 177F4010FE56CA3336300305F1656F24C74CD1D8
    - name: MARIADB_MAJOR
      value: "10.5"
    - name: GOSU_VERSION
      value: "1.12"
    - name: container
      value: podman
    - name: HOSTNAME
      value: myapp-pod
    image: docker.io/library/mariadb:10.5
    name: myapp-db
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
      seLinuxOptions: {}
    workingDir: /
status: {}
---
metadata:
  creationTimestamp: null
spec: {}
status:
  loadBalancer: {}

It seems podman does not allow pulling images from localhost. I tried changing some settings in /etc/containers/registries.conf like setting insecure=true and adding 127.0.0.1 to registries, but the issue was not solved.

[vrothberg]

Do you have a registry running on localhost?

[djnotes]

No. I don't think. I never run registries on localhost. I didn't do it with Docker. Is it necessary with Podman? On Mon, Sep 21, 2020 at 14:22, Valentin Rothberg<[email protected]> wrote: Do you have a registry running on localhost? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[vrothberg]

The yaml is referencing image: localhost/myapp:latest which is why Podman tries to pull the image.

I assume the image exists on disk, does it? Podman behaves like Kubernetes where images with the latest tag are always being pulled. Changing the tag to something other than latest should solve the issue.

[djnotes]

Seems like it worked. Thanks. Interesting to see just a latest tag was confusing Podman. On Mon, Sep 21, 2020 at 15:02, Valentin Rothberg<[email protected]> wrote: The yaml is referencing image: localhost/myapp:latest which is why Podman tries to pull the image. I assume the image exists on disk, does it? Podman behaves like Kubernetes where images with the latest tag are always being pulled. Changing the tag to something other than latest should solve the issue. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[vrothberg]

Thanks for checking!

@rhatdan @haircommander, I've seen frequent confusion around the :latest auto-pull behaviour. Can we log that on stderr to make it more obvious? I feel that debugging this behaviour can easily lure into rabbit holes.

[djnotes]

There is another problem maybe needing its own issue. The mariadb container started by the yaml exits with 1 with this error:

2020-09-21T13:57:40.406307Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.21) starting as process 1
2020-09-21T13:57:40.415413Z 0 [ERROR] [MY-010123] [Server] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!
2020-09-21T13:57:40.415474Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-09-21T13:57:40.416018Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.21) MySQL Community Server - GPL.

Tried with a different image like mysql, but got the same error. Also tried changing privileged to true, but still got the same error. When starting the container with the same settings with normal user using Podman, it works fine, but with play kube mypod.yaml it seems it needs to run as the root user(?!).

[vrothberg]

Does the yaml work in K8s?

[djnotes]

I don't have access to a K8s environment on Fedora. I will try it on Mac's Kubernetes and post the update on that.

[djnotes]

I found that the problem was using the command key in the yaml file under containers. The containers are executable per se, so we don't need command there. Removed that and a lot of useless environment variables and the issue is gone. However, I have problem mounting a volume which I suspect is due to Selinux.
Here's the error:

2020-09-21 21:24:21+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.5+maria~focal started.
2020-09-21 21:24:21+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-09-21 21:24:21+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.5+maria~focal started.
ls: cannot open directory '/docker-entrypoint-initdb.d/': Permission denied

How can I host-mount in the yaml without offending Selinux. Is there some option like Z to use somewhere in the YAML?

[vrothberg]

How can I host-mount in the yaml without offending Selinux. Is there some option like Z to use somewhere in the YAML?

@haircommander, do you know?

Note that I am going to close the issue as the initial issue has been resolved. But we can continue the conversation here.

[haircommander]

@djnotes if you want to mount a volume onto the host, you can specify a hostpath volume:
https://kubernetes.io/docs/concepts/storage/volumes/#hostpath

but that doesn't seem like the problem, as you don't have the volume specified in the yaml. that means this /docker-entrypoint-initdb.d/ should be in the rootfs of the image, and should have been labelled.

sanity check: have you tried disabling selinux and running to see if it's actually that?

[djnotes]

@haircommander sorry I forgot to update the code. The final yaml that has permission issue is this:

# Generation of Kubernetes YAML is still under development!
#
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-2.0.6
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2020-09-19T19:43:25Z"
  labels:
    app: myapp-pod
  name: myapp-pod
spec:
  containers:
  - name: myapp-db
    env:
    - name: MYSQL_PASSWORD
      value: ZsC_jDjJRW2H
    - name: MYSQL_ROOT_PASSWORD
      value: ZsC_jDjJRW2H
    - name: GOSU_VERSION
      value: "1.12"
    - name: MYSQL_USER
      value: myapp
    - name: MYSQL_DATABASE
      value: myapp
    - name: HOSTNAME
      value: myapp-pod
    image: docker.io/library/mariadb:10.5
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
      seLinuxOptions: {}
    volumeMounts:
    - mountPath: /docker-entrypoint-initdb.d
      name: home-johndoe-src-myapp-data
    workingDir: /
  # - command:
  #   - python3
  #   - app.py
  - name: myapp-app
    env:
    - name: HOSTNAME
      value: myapp-pod
    image: localhost/myapp
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
      seLinuxOptions: {
        level: ""
      }
    workingDir: /app
  volumes:
  - hostPath:
      path: /home/johndoe/src/myapp/data
      type: Directory
    name: home-johndoe-src-myapp-data
status: {}
---
metadata:
  creationTimestamp: null
spec: {}
status:
  loadBalancer: {}