Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Moby client behavior rely on Moby API, and is not compatible with podman short name handling #12320

Closed
mscherer opened this issue Nov 16, 2021 · 3 comments · Fixed by #12435
Closed

Moby client behavior rely on Moby API, and is not compatible with podman short name handling #12320

mscherer opened this issue Nov 16, 2021 · 3 comments · Fixed by #12435
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@mscherer
Copy link
Contributor

According to https://github.com/moby/moby/blob/master/client/image_pull.go#L22-L29 , "docker.io" suffix is stripped when a program call client.ImagePull (since that's the "familiar name")

For example, trying to client.ImagePull(_, "docker.io/a6543/test_git_plugin", _ ) will result in a API call of

POST /v1.33/images/create?fromImage=a6543%2Ftest_git_plugin&tag=latest

This is not compatible with the short-name feature of podman (https://www.redhat.com/sysadmin/container-image-short-names), since this result in podman returning this error in the API (tested with the HEAD version, see #12315 for the whole story):

Error response from daemon: failed to resolve image name: short-name resolution enforced but cannot prompt without a TTY

Should the compat API mirror moby client expectation, eg, defaulting to docker.io ?
@mscherer mscherer mentioned this issue Nov 16, 2021
Open
@mscherer mscherer changed the title Moby client API is not compatible with podman Moby client behavior rely on Moby API, and is not compatible with podman short name handling Nov 16, 2021
@mscherer mscherer mentioned this issue Nov 16, 2021
Closed
mscherer added a commit to mscherer/podman that referenced this issue Nov 16, 2021
@mscherer
Add docker.io to unqualified image name
5ad0f0d 
Since that's the default behavior of moby engine, the compat
API need to reflect that.

Fixes containers#12320

Signed-off-by: Michael Scherer <[email protected]>
vrothberg added a commit to vrothberg/image that referenced this issue Nov 23, 2021
@vrothberg
shortnames: mechanism to enforce resolving to Docker Hub
707359b 
Podman's Docker-compatible REST API is in need of a mechanism to enforce
resolving to Docker Hub only.  Yet there is the desire for the rest of
the stack to continue honoring the system's registries.conf.

We recently added a new field to containers.conf [1] which allows for
opting out from enforcing Docker Hub for Podman's compat API but we
still lack a way of enforcement when resolving short names; which
ultimately is *the* place to do that.

This change does the necessary plumbing.  The compat REST handlers will
set the new field in the `types.SystemContext` and pass that down to
libimage and buildah.

[1] containers/common@e698b8c

Context: containers/podman/issues/12320
Signed-off-by: Valentin Rothberg <[email protected]>
@vrothberg vrothberg mentioned this issue Nov 23, 2021
Merged
vrothberg added a commit to vrothberg/image that referenced this issue Nov 24, 2021
@vrothberg
shortnames: mechanism to enforce resolving to Docker Hub
af3cbe5 
Podman's Docker-compatible REST API is in need of a mechanism to enforce
resolving to Docker Hub only.  Yet there is the desire for the rest of
the stack to continue honoring the system's registries.conf.

We recently added a new field to containers.conf [1] which allows for
opting out from enforcing Docker Hub for Podman's compat API but we
still lack a way of enforcement when resolving short names; which
ultimately is *the* place to do that.

This change does the necessary plumbing.  The compat REST handlers will
set the new field in the `types.SystemContext` and pass that down to
libimage and buildah.

[1] containers/common@e698b8c

Context: containers/podman/issues/12320
Signed-off-by: Valentin Rothberg <[email protected]>
@vrothberg vrothberg mentioned this issue Nov 29, 2021
Merged
vrothberg added a commit to vrothberg/image that referenced this issue Nov 29, 2021
@vrothberg
shortnames: mechanism to enforce resolving to Docker Hub
cff8707 
Podman's Docker-compatible REST API is in need of a mechanism to enforce
resolving to Docker Hub only.  Yet there is the desire for the rest of
the stack to continue honoring the system's registries.conf.

We recently added a new field to containers.conf [1] which allows for
opting out from enforcing Docker Hub for Podman's compat API but we
still lack a way of enforcement when resolving short names; which
ultimately is *the* place to do that.

This change does the necessary plumbing.  The compat REST handlers will
set the new field in the `types.SystemContext` and pass that down to
libimage and buildah.

[1] containers/common@e698b8c

Context: containers/podman/issues/12320
Signed-off-by: Valentin Rothberg <[email protected]>
vrothberg added a commit to vrothberg/libpod that referenced this issue Nov 30, 2021
@vrothberg
compat API: allow enforcing short-names resolution to Docker Hub
5bdd571 
The Docker-compatible REST API has historically behaved just as the rest
of Podman and Buildah (and the atomic Docker in older RHEL/Fedora) where
`containers-registries.conf` is centrally controlling which registries
a short name may resolve to during pull or local image lookups.  Please
refer to a blog for more details [1].

Docker, however, is only resolving short names to docker.io which has
been reported (see containers#12320) to break certain clients who rely on this
behavior.  In order to support this scenario, `containers.conf(5)`
received a new option to control whether Podman's compat API resolves
to docker.io only or behaves as before.

Most endpoints allow for directly normalizing parameters that represent
an image.  If set in containers.conf, Podman will then normalize the
references directly to docker.io.  The build endpoint is an outlier
since images are also referenced in Dockerfiles.  The Buildah API,
however, supports specifying a custom `types.SystemContext` in which
we can set a field that enforces short-name resolution to docker.io
in `c/image/pkg/shortnames`.

Notice that this a "hybrid" approach of doing the normalization directly
in the compat endpoints *and* in `pkg/shortnames` by passing a system
context.  Doing such a hybrid approach is neccessary since the compat
and the libpod endpoints share the same `libimage.Runtime` which makes
a global enforcement via the `libimage.Runtime.systemContext`
impossible.  Having two separate runtimes for the compat and the libpod
endpoints seems risky and not generally applicable to all endpoints.

[1] https://www.redhat.com/sysadmin/container-image-short-names

Fixes: containers#12320
Signed-off-by: Valentin Rothberg <[email protected]>
nlewo pushed a commit to nlewo/image that referenced this issue Dec 26, 2021
@vrothberg @nlewo
shortnames: mechanism to enforce resolving to Docker Hub
d703342 
Podman's Docker-compatible REST API is in need of a mechanism to enforce
resolving to Docker Hub only.  Yet there is the desire for the rest of
the stack to continue honoring the system's registries.conf.

We recently added a new field to containers.conf [1] which allows for
opting out from enforcing Docker Hub for Podman's compat API but we
still lack a way of enforcement when resolving short names; which
ultimately is *the* place to do that.

This change does the necessary plumbing.  The compat REST handlers will
set the new field in the `types.SystemContext` and pass that down to
libimage and buildah.

[1] containers/common@e698b8c

Context: containers/podman/issues/12320
Signed-off-by: Valentin Rothberg <[email protected]>
@Luap99 Luap99 mentioned this issue Feb 15, 2022
Closed
@petersenna
Copy link

petersenna commented May 19, 2022 •
edited
Loading

@vrothberg I am not sure I ran into a corner case, so I do not really know if the behavior I describe is a bug or not. For reasons I pull container images from a private registry at VM image creation time, and then after VMs are deployed I create containers over the Podman's Docker-compatible REST API.

The issue I am having is that starting with podman 4, trying to start a container that was previously pulled fails because the container does not exist at Docker Hub. More precisely I get a 404 for docker.io/library/NAME:TAG

Here is an example. Please note that I replaced real strings with words in capital letters, like NAME, and TAG.

# podman --version
podman version 4.0.2

# podman --log-level=debug system service -t 0 tcp:0.0.0.0:2376
...
DEBU[0017] Looking up image "NAME:TAG" in local containers storage
DEBU[0017] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0017] Trying "NAME:TAG" ...
DEBU[0017] Looking up image "NAME:TAG" in local containers storage
DEBU[0017] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0017] Trying "NAME:TAG" ...
DEBU[0017] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0017] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0017] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf"
DEBU[0017] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
DEBU[0017] Trying "localhost/NAME:TAG" ...
DEBU[0017] Trying "registry.fedoraproject.org/NAME:TAG" ...
DEBU[0017] Trying "registry.access.redhat.com/NAME:TAG" ...
DEBU[0017] Trying "registry.centos.org/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "localhost/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "registry.fedoraproject.org/NAME:TAG" ...
DEBU[0017] Trying "registry.access.redhat.com/NAME:TAG" ...
DEBU[0017] Trying "registry.centos.org/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "REGISTRY/PATH/NAME:TAG" ...
DEBU[0017] Trying "REGISTRY/PATH/NAME:TAG" ...
DEBU[0017] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5fb70826285b943f31f0c4b0dc0247d29177941c6e01a99b188bdda633244892"
DEBU[0017] Found image "NAME:TAG" as "REGISTRY/PATH/NAME:TAG" in local containers storage
DEBU[0017] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5fb70826285b943f31f0c4b0dc0247d29177941c6e01a99b188bdda633244892"
DEBU[0017] Found image "NAME:TAG" as "REGISTRY/PATH/NAME:TAG" in local containers storage
DEBU[0017] Found image "NAME:TAG" as "REGISTRY/PATH/NAME:TAG" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5fb70826285b943f31f0c4b0dc0247d29177941c6e01a99b188bdda633244892)
DEBU[0017] Looking up image "docker.io/library/NAME:TAG" in local containers storage
DEBU[0017] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Found image "NAME:TAG" as "REGISTRY/PATH/NAME:TAG" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5fb70826285b943f31f0c4b0dc0247d29177941c6e01a99b188bdda633244892)
DEBU[0017] Looking up image "docker.io/library/NAME:TAG" in local containers storage
DEBU[0017] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
INFO[0017] Request Failed(Not Found): No such image: docker.io/library/NAME:TAG: image not known
DEBU[0017] Trying "docker.io/library/NAME:TAG" ...
INFO[0017] Request Failed(Not Found): No such image: docker.io/library/NAME:TAG: image not known

Then if I set compat_api_enforce_docker_hub to false, podman works as I would expect and will be happy with containers found in local containers storage.

Is this a bug?

P.S. On the VMs I have running docker, Docker is perfectly fine with containers found in local containers storage.

@vrothberg
Copy link
Member

Thanks for the ping, @petersenna. It definitely sounds suspicious when Docker works but Podman does not on the compat API and by definition qualifies as a bug.

Judging from the logs, Podman is able to resolve the image locally when looking for "NAME:TAG":

DEBU[0017] Looking up image "NAME:TAG" in local containers storage
[...]
DEBU[0017] Found image "NAME:TAG" as "REGISTRY/PATH/NAME:TAG [...]

For some reasons, Podman is then looking up the normalized version which fails:

DEBU[0017] Looking up image "docker.io/library/NAME:TAG" in local containers storage
[...]
INFO[0017] Request Failed(Not Found): No such image: docker.io/library/NAME:TAG: image not known

Would you mind opening a new issue for it? It would be extremely useful to see the exact API request to isolate the bug.

@petersenna petersenna mentioned this issue May 19, 2022
Closed
@petersenna
Copy link

Created a new issue: #14291

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
3 participants
@mscherer @petersenna @vrothberg