[ECR] [Bug]: Creation of two untagged images when pulling an image through ECR Cache

[sumanthst25]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
I had observed a bug with pulling images using ECR Cache

Which service(s) is this request for?
ECR

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
The issue is when I pull an image from the upstream repository in the public registry using ECR cache it creates two additional untagged images along with the image with the tag mentioned in the "docker pull" command.

Are you currently working around this issue?
How are you currently solving this problem?
I am currently implementing life cycle policies for the repositories to delete un tagged images
https://docs.aws.amazon.com/AmazonECR/latest/userguide/LifecyclePolicies.html

Additional context
Anything else we should know?
Below is my replication scenario for more understanding of the issue :

$ docker pull .dkr.ecr..amazonaws.com/ecr-public/amazonlinux/amazonlinux:latest
latest: Pulling from ecr-public/amazonlinux/amazonlinux
xxx: Pull complete
Digest: sha256:xxxxxxxxxxxxxxx
Status: Downloaded newer image for .dkr.ecr..amazonaws.com/ecr-public/amazonlinux/amazonlinux:latest
.dkr.ecr..amazonaws.com/ecr-public/amazonlinux/amazonlinux:latest

$ aws ecr list-images --repository-name ecr-public/amazonlinux/amazonlinux --filter tagStatus=UNTAGGED --region
{
"imageIds": [
{
"imageDigest": "sha256:xxxxxxxxxxxx"
},
{
"imageDigest": "sha256:xxxxxxxxxxxx"
}
]
}

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[Vlaaaaaaad]

This may actually be the display bug for the ECR Console, with a fix requested in #1596.

When a multi-arch image is pushed to ECR, like amazonlinux:latest (which has two architectures/versions -- one for x86_64 AKA Intel and one for arm64 AKA Graviton), three images appear in the ECR Repo: latest which is actually a manifest, and then 2 untagged images which are the x86_64 and arm64 images.

The ECR cache may be pulling both images to your private ECR repo and that's why the console shows 3 images.

[guvmao]

After a long time, is there still no progress on this issue?

[dariocurr]

Has anyone found a solution to have only a tagged image in the ECR registry?

[ktomaszx]

I think I'm struggling with the same issue. I have to use lifecycle policy to delete untagged images. They are produced when using caching mechanism and size of image is the same as tagged image. In my case exporting cache creates second untagged image.

[jlbutler]

Hi everyone, apologies for such a long delay replying to this issue. I've just put an update on the issue tagged above, which I hope is helpful.

#1596 (comment)

I don't think there is a bug here, but definitely a UX improvement we can make. The ECR console shows all content - images, artifacts, indexes, etc. And we show it whether it's tagged or untagged content. So in this case, a multiplatform image with two supported ISAs shows up as three things in the view: one index and two images. Whether the underlying images are tagged or not is up to you - using BuildKit image are by default untagged.

All this said, we would like to make this a nicer and more obvious experience. We are considering making a change which would nest the related images under the index in the view, so you would by default see the tagged index, and then click to expand and see the referred images. We hope this will clarify things and make it easier to reason about.

I'm going to close this one as a duplicate of #1596 since this isn't a bug related to PTC, but an enhancement for a better console view. Thanks!

[carocad]

@jlbutler coming back to the creation of 2 images on ECR when pushing a multiplatform one.

I think there is indeed a bug on ECR, not for the lifecycle as mentioned in the linked comments but rather due to mutability (or lack thereof) of the tags.

In my case, our repositories have immutable tags, so when trying to push a multiplatform image to ECR it fails with a failed commit on ref unexpected status from put .... Checking the dockerd debug log I found out that the 2 <untagged> images are the culprit as both have the "same tag" so ECR rejects pushing it twice.

[carocad]

probably related to moby/buildkit#2004 even though the linked PR over there says that the issue has been solved. In my case I am using containerd for building the multiplatform images not buildx with docker-container.