Build image `pull` field type mismatch with Docker #17778

cristianrgreco · 2023-03-14T13:07:20Z

Issue Description

Docker accepts any string value for the pull field to indicate the image should be pulled:https://docs.docker.com/engine/api/v1.42/#tag/Image/operation/ImageBuild

Podman expects the value to equal either boolean true or string true:
https://docs.podman.io/en/latest/_static/api.html?version=v4.4#tag/images/operation/ImageBuildLibpod

Applications that align with Docker's spec are unable to switch to Podman as building an image with this field fails.

Steps to reproduce the issue

POST to the create image endpoint with pull set to a string, such as always.

Describe the results you received

No response stream received

Describe the results you expected

Expected a response stream

podman info output

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.7-0debian9999+obs15.6_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 87.46
    systemPercent: 5.69
    userPercent: 6.84
  cpus: 2
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: fv-az646-90
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 123
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.15.0-1034-azure
  linkmode: dynamic
  logDriver: journald
  memFree: 4857856000
  memTotal: 7281278976
  networkBackend: netavark
  ociRuntime:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/runner/.local/share/containers/storage
  graphRootAllocated: 89297309696
  graphRootUsed: 58336636928
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /home/runner/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.2

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock

Additional information

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2023-04-14T00:06:11Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2023-04-14T11:28:34Z

Do you have an example with curl to show us failing, where docker succeeds?

eriksjolund · 2023-05-06T08:39:58Z

Reproduced on Fedora CoreOS (fedora-coreos-38.20230430.1.0-qemu.aarch64)

boot up a new Fedora CoreOS VM
sudo useradd test
sudo usermod -aG docker test
sudo systemctl start docker.service
Open an interactive shell session for user test
```
sudo machinectl shell test@`
```
mkdir dir && cd dir

Create a file

cat << EOF > Dockerfile
FROM docker.io/library/alpine
EOF

systemctl --user start podman.socket

Run command

tar cf - . | curl -XPOST \
  --unix-socket /var/run/docker.sock \
  --data-binary @- \
  -H 'Content-Type: application/x-tar' \
  http://localhost/v1.41/build?pull=always

Run command

tar cf - . | curl -XPOST \
  --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock \
  --data-binary @- \
  -H 'Content-Type: application/x-tar' \
  http://localhost/v1.41/build?pull=always

Run command

tar cf - . | curl -XPOST \
  --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock \
  --data-binary @- \
  -H 'Content-Type: application/x-tar' \
  http://localhost/v1.41/build?pull=true

Output from step 9

{"stream":"Step 1/1 : FROM docker.io/library/alpine"}
{"stream":"\n"}
{"status":"Pulling from library/alpine","id":"latest"}
{"status":"Pulling fs layer","progressDetail":{},"id":"c41833b44d91"}
{"status":"Downloading","progressDetail":{"current":32768,"total":3261854},"progress":"[\u003e                                                  ]  32.77kB/3.262MB","id":"c41833b44d91"}
{"status":"Verifying Checksum","progressDetail":{},"id":"c41833b44d91"}
{"status":"Download complete","progressDetail":{},"id":"c41833b44d91"}
{"status":"Extracting","progressDetail":{"current":32768,"total":3261854},"progress":"[\u003e                                                  ]  32.77kB/3.262MB","id":"c41833b44d91"}
{"status":"Extracting","progressDetail":{"current":294912,"total":3261854},"progress":"[====\u003e                                              ]  294.9kB/3.262MB","id":"c41833b44d91"}
{"status":"Extracting","progressDetail":{"current":884736,"total":3261854},"progress":"[=============\u003e                                     ]  884.7kB/3.262MB","id":"c41833b44d91"}
{"status":"Extracting","progressDetail":{"current":2949120,"total":3261854},"progress":"[=============================================\u003e     ]  2.949MB/3.262MB","id":"c41833b44d91"}
{"status":"Extracting","progressDetail":{"current":3261854,"total":3261854},"progress":"[==================================================\u003e]  3.262MB/3.262MB","id":"c41833b44d91"}
{"status":"Pull complete","progressDetail":{},"id":"c41833b44d91"}
{"status":"Digest: sha256:124c7d2707904eea7431fffe91522a01e5a861a624ee31d03372cc1d138a3126"}
{"status":"Status: Downloaded newer image for alpine:latest"}
{"stream":" ---\u003e 51e60588ff2c\n"}
{"aux":{"ID":"sha256:51e60588ff2cd9f45792b23de89bfface0a7fbd711d17c5f5ce900a4f6b16260"}}
{"stream":"Successfully built 51e60588ff2c\n"}

There was no output from step 10

Output from step 11

{"stream":"STEP 1/1: FROM docker.io/library/alpine\n"}
{"stream":"Trying to pull docker.io/library/alpine:latest...\n"}
{"stream":"Getting image source signatures\n"}
{"stream":"Copying blob sha256:c41833b44d910632b415cd89a9cdaa4d62c9725dc56c99a7ddadafd6719960f9\n"}
{"stream":"Copying config sha256:51e60588ff2cd9f45792b23de89bfface0a7fbd711d17c5f5ce900a4f6b16260\n"}
{"stream":"Writing manifest to image destination\n"}
{"stream":"Storing signatures\n"}
{"stream":"COMMIT\n"}
{"stream":"--\u003e 51e60588ff2c\n"}
{"stream":"51e60588ff2cd9f45792b23de89bfface0a7fbd711d17c5f5ce900a4f6b16260\n"}
{"aux":{"ID":"sha256:51e60588ff2cd9f45792b23de89bfface0a7fbd711d17c5f5ce900a4f6b16260"}}
{"stream":"Successfully built 51e60588ff2c\n"}

About the system:

[test@localhost ~]$ rpm -q podman
podman-4.5.0-1.fc38.aarch64
[test@localhost ~]$ rpm -q moby-engine
moby-engine-20.10.23-1.fc38.aarch64
[test@localhost ~]$

`pull` parameter in `build` must accept string just like docker. Ref: https://docs.docker.com/engine/api/v1.42/#tag/Image/operation/ImageBuild Closes: containers#17778 Signed-off-by: Aditya R <[email protected]>

matejvasek · 2023-08-10T21:17:05Z

@eriksjolund @cristianrgreco @flouthoc

This is really strange. The docker client sets this value to "1" (for true), and server handles it like a boolean too.

client:
https://github.com/moby/moby/blob/e0da5cb9291982a4c7b3c6b0642eeced4ba9f8ac/client/image_build.go#L71-L73

server:
https://github.com/moby/moby/blob/f71fe8476a2f8493adf50b6d520a739568383a51/api/server/router/build/build_routes.go#L77-L79
https://github.com/moby/moby/blob/master/api/server/httputils/form.go#L14-L16

This means the always string is not interpreted as an pull-policy enum value but rather as boolean true (but that is rather accident).
Actually it appears server consider everything but "", "0", "no", "false", "none" to be true (+- case).

The PR #18583 actually breaks compatibility instead of fixing it.

# Broken compatibility with the docker CLI
docker build . -t alpine-1000 --pull
Sending build context to Docker daemon  2.048kB
Error response from daemon: failed to parse query parameter 'pull': "1": invalid pull policy: "1"

I would suggest to revert it back to boolean. The older code is not 100% compliant but much better that what we have now. Then it could be improved upon by register custom boolean deserializer for compat endpoints.

	dec.RegisterConverter(true, func(s string) reflect.Value {
		s = strings.ToLower(strings.TrimSpace(s))
		return reflect.ValueOf(!(s == "" || s == "0" || s == "no" || s == "false" || s == "none"))
	})

You may argue that doc says it's a string but I trust actual code much more.
Also it does not specify format of the string.

matejvasek · 2023-08-10T21:29:22Z

IMO the Docker API documentation should be fixed.

matejvasek · 2023-08-11T02:30:49Z

Looking at the source code of Docker it seems there is not such thing as pull policy. It simply interprets Always == true but note that also Never == true.

This reverts commit 5b148a0. Reverting to treating the `pull` query parameter as a boolean. Because of deceiving Docker API documentation it was assumed that the parameter is pull-policy, however that is not true. Docker does treat `pull` as a boolean. What is interesting is that Docker indeed accepts strings like `always` or `never` however Docekr both of these strings treat as `true`, not as pull-policy. As matter of the fact it seems there is no such a thing as pull-policy in Docker. More context containers#17778 (comment) Signed-off-by: Matej Vasek <[email protected]>

cristianrgreco added the kind/bug Categorizes issue or PR as related to a bug. label Mar 14, 2023

cristianrgreco mentioned this issue Mar 14, 2023

Fix issue with DOCKER_HOST and add limited Podman CI workflow testcontainers/testcontainers-node#494

Merged

github-actions bot added the stale-issue label Apr 14, 2023

rhatdan added HTTP API Bug is in RESTful API Good First Issue This issue would be a good issue for a first time contributor to undertake. and removed stale-issue labels Apr 14, 2023

flouthoc mentioned this issue May 16, 2023

compat,build: pull must accept string #18583

Merged

openshift-merge-robot closed this as completed in #18583 May 16, 2023

matejvasek mentioned this issue Aug 10, 2023

Fix pull parameter parsing for compat /build endpoint #19591

Merged

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Nov 10, 2023

github-actions bot locked as resolved and limited conversation to collaborators Nov 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build image `pull` field type mismatch with Docker #17778

Build image `pull` field type mismatch with Docker #17778

cristianrgreco commented Mar 14, 2023 •

edited

Loading

github-actions bot commented Apr 14, 2023

rhatdan commented Apr 14, 2023

eriksjolund commented May 6, 2023 •

edited

Loading

matejvasek commented Aug 10, 2023 •

edited

Loading

matejvasek commented Aug 10, 2023 •

edited

Loading

matejvasek commented Aug 11, 2023 •

edited

Loading

Build image pull field type mismatch with Docker #17778

Build image pull field type mismatch with Docker #17778

Comments

cristianrgreco commented Mar 14, 2023 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

github-actions bot commented Apr 14, 2023

rhatdan commented Apr 14, 2023

eriksjolund commented May 6, 2023 • edited Loading

matejvasek commented Aug 10, 2023 • edited Loading

matejvasek commented Aug 10, 2023 • edited Loading

matejvasek commented Aug 11, 2023 • edited Loading

Build image `pull` field type mismatch with Docker #17778

Build image `pull` field type mismatch with Docker #17778

cristianrgreco commented Mar 14, 2023 •

edited

Loading

eriksjolund commented May 6, 2023 •

edited

Loading

matejvasek commented Aug 10, 2023 •

edited

Loading

matejvasek commented Aug 10, 2023 •

edited

Loading

matejvasek commented Aug 11, 2023 •

edited

Loading