docker-java push fails against the podman service

[Hendrik-H]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. start podman as a system service: podman system service tcp:0.0.0.0:2375 --timo=0

2. wrote a gradle build script that uses https://github.com/bmuschko/gradle-docker-plugin to build docker image and to push it

3. the build fails during the push

Describe the results you received:
The push fails with:

Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not push image
	at com.github.dockerjava.core.command.PushImageResultCallback.throwFirstError(PushImageResultCallback.java:40)
	at com.github.dockerjava.api.async.ResultCallbackTemplate.awaitCompletion(ResultCallbackTemplate.java:93)
	at com.bmuschko.gradle.docker.tasks.image.DockerPushImage.runRemoteCommand(DockerPushImage.groovy:63)
	at com.bmuschko.gradle.docker.tasks.AbstractDockerRemoteApiTask.start(AbstractDockerRemoteApiTask.groovy:74)

Describe the results you expected:
The push to work as podman's API implementation should be compatible to that of docker.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 14:38:17 2020
OS/Arch:      linux/s390x

Output of podman info --debug:

host:
  arch: s390x
  buildahVersion: 1.18.0
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-3.fc33.s390x
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: d47b0c4859cb71c991115d47c69e732fe8e5d30c'
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: file
  hostname: podman-host-0
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-147.el8.s390x
  linkmode: dynamic
  memFree: 401321984
  memTotal: 16868089856
  ociRuntime:
    name: crun
    package: crun-0.16-1.fc33.s390x
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 338h 44m 58.82s (Approximately 14.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.3.0-1.fc33.s390x
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.3
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438297
  BuiltTime: Tue Dec  8 14:38:17 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/s390x
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.2.1-1.fc33.s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
A pod in an OpenShift setup.

[Hendrik-H]

I debugged the issue a bit further. The problem is that the "podman push" REST call does not return any data, expect a line feed character, while docker returns a stream of JSON status object like:

{"status":"The push refers to repository [test-registry/test/test-image]"}
{"status":"Preparing","progressDetail":{},"id":"f869c1b43480"}
{"status":"Preparing","progressDetail":{},"id":"514c3a3e64d4"}
{"status":"Pushing","progressDetail":{"current":34816,"total":2196641},"progress":"[\u003e                                                  ]  34.82kB/2.197MB","id":"f869c1b43480"}
{"status":"Pushing","progressDetail":{"current":133120,"total":2196641},"progress":"[===\u003e                                               ]  133.1kB/2.197MB","id":"f869c1b43480"}
{"status":"Pushing","progressDetail":{"current":919552,"total":2196641},"progress":"[====================\u003e                              ]  919.6kB/2.197MB","id":"f869c1b43480"}
{"status":"Layer already exists","progressDetail":{},"id":"514c3a3e64d4"}
{"status":"Pushing","progressDetail":{"current":1247232,"total":2196641},"progress":"[============================\u003e                      ]  1.247MB/2.197MB","id":"f869c1b43480"}
{"status":"Pushing","progressDetail":{"current":1771520,"total":2196641},"progress":"[========================================\u003e          ]  1.772MB/2.197MB","id":"f869c1b43480"}
{"status":"Pushing","progressDetail":{"current":1968128,"total":2196641},"progress":"[============================================\u003e      ]  1.968MB/2.197MB","id":"f869c1b43480"}
{"status":"Pushing","progressDetail":{"current":2164736,"total":2196641},"progress":"[=================================================\u003e ]  2.165MB/2.197MB","id":"f869c1b43480"}
{"status":"Pushing","progressDetail":{"current":2200064,"total":2196641},"progress":"[==================================================\u003e]    2.2MB","id":"f869c1b43480"}
{"status":"Pushed","progressDetail":{},"id":"f869c1b43480"}
{"status":"1: digest: sha256:0b65651726f2f0209c5f8ff21f782a8e783fecb05ee2c5b57433ad516fdb0696 size: 738"}
{"progressDetail":{},"aux":{"Tag":"1","Digest":"sha256:0b65651726f2f0209c5f8ff21f782a8e783fecb05ee2c5b57433ad516fdb0696","Size":738}}

An older version of the docker REST API shows the format: https://docs.docker.com/engine/api/v1.24/

HTTP/1.1 200 OK
Content-Type: application/json

{"status": "Pushing..."}
{"status": "Pushing", "progress": "1/? (n/a)", "progressDetail": {"current": 1}}}
{"error": "Invalid..."}
...

In the latest version I can however not see that part: https://docs.docker.com/engine/api/v1.41/

Anyhow, the difference in the returned data format is breaking the clients and thus the Docker REST API support of Podman.

[Hendrik-H]

The HTTP API entry point for the push seems to be here: https://github.com/containers/podman/blob/master/pkg/api/handlers/compat/images_push.go#L18
And the problem should start here: https://github.com/containers/podman/blob/master/pkg/api/handlers/compat/images_push.go#L72
The code just passes in stderr for the progress reporting and then returns the success just based on the status code. I followed the push logic up to

podman/vendor/github.com/containers/image/v5/copy/copy.go

Line 222 in 0fd31e2

reportWriter := ioutil.Discard

To implement this in a compatible way a problem should be on what point the JSON objects need to be created. Logical seems to do this on the HTTP side but then a simple Writer as the progress reporting interface does not work. Figuring out on the otherside that this is about JSON seems a bit ugly but would allow the HTTP side to just read out the progress lines and send them blindly to the caller.

A small thing that looks odd to me is: https://github.com/containers/podman/blob/master/pkg/api/handlers/compat/images_push.go#L66
As the API spec states "The push is cancelled if the HTTP connection is closed." I would have expected that the passed on Context is canceled when the request is canceled but that does not seem to be the case here.

[mheon]

@jwhonce PTAL

[Hendrik-H]

Hi, any chance somebody looked into this yet? This prevents us from switching from docker to podman.

[rhatdan]

@Hendrik-H Sorry I don't believe anyone has looked at it.

[jwhonce]

@Hendrik-H Can you share your gradle script?

[Hendrik-H]

Hi @jwhonce ,

here is a sample build.gradle.kts file:

import com.bmuschko.gradle.docker.tasks.image.*

plugins {
    id("com.bmuschko.docker-remote-api") version "6.6.1"
}

docker {
    url.set("tcp://127.0.0.1:2375")
}

val createDockerfile by tasks.registering(Dockerfile::class) {
    from("busybox:stable")

    entryPoint("echo", "hello world")
}

val dockerBuild by tasks.registering(DockerBuildImage::class) {
    dependsOn(createDockerfile)

    images.add("localhost:5000/push-test:latest")
}

val dockerPush by tasks.registering(DockerPushImage::class) {
    dependsOn(dockerBuild)

    images.add("localhost:5000/push-test:latest")
}

For the test I started podman as a docker replacement using:

podman system service --log-level debug --time=0 tcp:127.0.0.1:2375

I also started a local docker registry using:

podman run -it -p 5000:5000 --rm --name registry registry:2

Lastly I edited /etc/containers/registries.conf to make the test registry insecure.

The podman log shows that the push was actually successful but as stated the Java client that is used under the covers fails as the REST call to podman does not return any status JSON but just a 200.

[mlegenovic]

The same problem is described in issue #7857, just for pull endpoint. Can be reproduced with:
curl -X POST 'http://localhost:8080/images/create?fromImage=docker.io%2Flibrary%2Fmysql&tag=latest'

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mlegenovic Does this issue still exists? I thought we had fixed all of the Java issues.

[mlegenovic]

@rhatdan I forgot this one :-( too many things in my head. I had it locally on my PC, but forgot to create PR.