Error trying to push a manifest to azure registry with retention policy #1382

side-chains · 2024-05-16T08:58:19Z

What happened in your environment?

When trying to push a manifest with a tag to an azure registry with a retention policy, oras reports a 500: Internal Server Error. If I disable the retention policy on azure, I can push the tag successfully.

Interestingly, notation is able to push its blobs and manifests to the same registry, even when the retention policy is enabled. This suggests that there is something that can be done on the client side to make it work.

Notes:

Pushing the manifest without the tag works correctly,

cat manifest | oras manifest push -d -v my-registry.azurecr.io/my/repo -

Pushing the manifest with a tag produces the output below,

cat manifest | oras manifest push -d -v my-registry.azurecr.io/my/repo:my-tag -

Pushing the manifest without a tag, and then tagging with oras tag also produces a 500 error.

The detailed output, with -d -v flags (I have added separator lines for visual comfort):

——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:28+02:00 level=debug msg=Request #0
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #0
< Response Status: "401 Unauthorized"
< Response headers:
   "Www-Authenticate": "Bearer realm=\"https://my-registry.azurecr.io/oauth2/token\",service=\"my-registry.azurecr.io\",scope=\"repository:my/repo:pull\""
   "X-Content-Type-Options": "nosniff"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
   "Server": "AzureContainerRegistry"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "X-Ms-Correlation-Request-Id": "46c27317-b52e-4c52-a266-455d332d6646"
   "Content-Length": "217"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #1
> Request URL: "https://my-registry.azurecr.io/oauth2/token"
> Request method: "POST"
> Request headers:
   "Content-Type": "application/x-www-form-urlencoded"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #1
< Response Status: "200 OK"
< Response headers:
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "X-Ms-Correlation-Request-Id": "a68cf443-1d5f-4e66-8f5b-ff28eb9188f2"
   "X-Ms-Ratelimit-Remaining-Calls-Per-Second": "333.316667"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains"
   "Server": "AzureContainerRegistry"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #2
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #2
< Response Status: "404 Not Found"
< Response headers:
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
   "X-Ms-Request-Id": "704e3e35-17f4-4667-9f23-78efa2de3a00"
   "Server": "AzureContainerRegistry"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "X-Ms-Correlation-Request-Id": "c09c0f21-233b-49ce-b8d1-885f73c533bd"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Content-Length": "276"
   "X-Content-Type-Options": "nosniff"
   "X-Ms-Client-Request-Id": ""
Uploading 77f69ad47d16 application/vnd.oci.image.index.v1+json
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #3
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "PUT"
> Request headers:
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #3
< Response Status: "401 Unauthorized"
< Response headers:
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Server": "AzureContainerRegistry"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
   "X-Content-Type-Options": "nosniff"
   "X-Ms-Correlation-Request-Id": "21cc6879-d24f-4b6e-a7e3-e7d2bc932780"
   "Content-Length": "288"
   "Www-Authenticate": "Bearer realm=\"https://my-registry.azurecr.io/oauth2/token\",service=\"my-registry.azurecr.io\",scope=\"repository:my/repo:pull,push\""
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #4
> Request URL: "https://my-registry.azurecr.io/oauth2/token"
> Request method: "POST"
> Request headers:
   "Content-Type": "application/x-www-form-urlencoded"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #4
< Response Status: "200 OK"
< Response headers:
   "X-Ms-Ratelimit-Remaining-Calls-Per-Second": "333.3"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains"
   "Server": "AzureContainerRegistry"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "X-Ms-Correlation-Request-Id": "19fe5024-c9be-47c0-ad6f-cb7d28982426"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #5
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "PUT"
> Request headers:
   "User-Agent": "oras/1.1.0"
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "Authorization": "*****"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:51+02:00 level=debug msg=Response #5
< Response Status: "500 Internal Server Error"
< Response headers:
   "Content-Length": "0"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Ms-Client-Request-Id": ""
   "X-Ms-Request-Id": "7f778f02-d3c0-4206-aec8-69ad501a7d4a"
   "Server": "AzureContainerRegistry"
   "Connection": "keep-alive"
   "X-Content-Type-Options": "nosniff"
   "X-Ms-Correlation-Request-Id": "07bea16f-8787-4ed8-8f16-d06aefe63ca7"
   "Date": "Mon, 13 May 2024 21:07:51 GMT"
   "Content-Type": "application/json; charset=utf-8"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
Error: failed to tag my-tag: PUT "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag": response status code 500: Internal Server Error

What did you expect to happen?

A successful response when pushing the tag.

How can we reproduce it?

Setup an azure registry and enable a retention policy. Then run

cat manifest | oras manifest push -d -v my-registry.azurecr.io/my/repo:my-tag -

What is the version of your ORAS CLI?

Version:        1.1.0
Go version:     go1.21.0
Git commit:     7079c468a06fb5815c99395eb4aaf46dd459d3fa
Git tree state: clean

What is your OS environment?

ubuntu 22.04

Are you willing to submit PRs to fix it?

Yes, I am willing to fix it.

The text was updated successfully, but these errors were encountered:

qweeah · 2024-05-17T08:32:01Z

@side-chains How is the retention policy setup? What is the retention period?

qweeah · 2024-05-17T08:33:22Z

I have added separator lines for visual comfort

@FeynmanZhou A strong plus on adding such visual comfort to the HTTP trace logs.

side-chains · 2024-05-17T14:38:20Z

@side-chains How is the retention policy setup? What is the retention period?

Click on the registry resource. On the left menu, under policies, there is an item named retention (preview). There you can only enable or disable the retention policy and, if it is enabled, you can choose the retention period in days.

I have tried with registries with retention periods of 5 and 7 days. The problem is the same in both cases.

qweeah · 2024-05-20T13:29:45Z

Checked registry logs and it should be a bug in ACR's retention feature. @side-chains Since you already have an Azure subscription, you can file a service ticket referencing this GitHub issue and we will follow it up in the ticket for privacy purposes.

side-chains · 2024-05-21T08:28:35Z

Checked registry logs and it should be a bug in ACR's retention feature. @side-chains Since you already have an Azure subscription, you can file a service ticket referencing this GitHub issue and we will follow it up in the ticket for privacy purposes.

@qweeah Does oras use the same base code as notation? Or, does it not interact with the registry in a similar way? If the bug is on azure's side, how does it explain that notation can push and tag without getting the error?

qweeah · 2024-05-21T08:40:57Z

As far as I know, notation should generate artifact without tags.

side-chains · 2024-05-21T08:49:59Z

As far as I know, notation should generate artifact without tags.

@qweeah Well, it creates a signature manifest, with the subject being the image that is being signed, and one or more layers with signatures. This one is, indeed, untagged.

However, it also creates an index manifest with a list of manifests pointing to all signature manifests corresponding to that image. This one is tagged as sha256- followed by the sha256 of the image that is being signed.

qweeah · 2024-05-21T09:02:04Z

Okay, in that case(referrers tag schema) ORAS and notation should use the same SDK, oras-go. Will check the implementation detail later. Back to your issue, I am pretty sure that errors are caused by ACR but cannot share server-side logs here for privacy consideration.

qweeah · 2024-05-28T06:08:34Z

@side-chains I have confirmed that both Notation and ORAS push manifests in the same way. Your error is caused by a known issue in ACR retention policy feature(I cannot share server-side logs here but feel free to join our slack channel https://cloud-native.slack.com/archives/CJ1KHJM5Z to discuss). I would suggest you raise a ticket and get help from ACR supports directly.

sajayantony · 2024-06-10T19:54:11Z

@FeynmanZhou one more usecase IMHO remove -d and linking - #911

github-actions · 2024-08-10T01:53:25Z

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

github-actions · 2024-09-09T01:59:00Z

This issue was closed because it has been stalled for 30 days with no activity.

side-chains added bug Something isn't working triage New issues or PRs to be acknowledged by maintainers labels May 16, 2024

qweeah removed the triage New issues or PRs to be acknowledged by maintainers label May 17, 2024

qweeah added Azure Contain Registry Azure Contain Registry related issues and removed bug Something isn't working labels May 21, 2024

github-actions bot added the stale Inactive issues or pull requests label Aug 10, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error trying to push a manifest to azure registry with retention policy #1382

Error trying to push a manifest to azure registry with retention policy #1382

side-chains commented May 16, 2024 •

edited

Loading

qweeah commented May 17, 2024

qweeah commented May 17, 2024

side-chains commented May 17, 2024

qweeah commented May 20, 2024 •

edited

Loading

side-chains commented May 21, 2024 •

edited

Loading

qweeah commented May 21, 2024

side-chains commented May 21, 2024

qweeah commented May 21, 2024

qweeah commented May 28, 2024 •

edited

Loading

sajayantony commented Jun 10, 2024 •

edited

Loading

github-actions bot commented Aug 10, 2024

github-actions bot commented Sep 9, 2024

Error trying to push a manifest to azure registry with retention policy #1382

Error trying to push a manifest to azure registry with retention policy #1382

Comments

side-chains commented May 16, 2024 • edited Loading

What happened in your environment?

What did you expect to happen?

How can we reproduce it?

What is the version of your ORAS CLI?

What is your OS environment?

Are you willing to submit PRs to fix it?

qweeah commented May 17, 2024

qweeah commented May 17, 2024

side-chains commented May 17, 2024

qweeah commented May 20, 2024 • edited Loading

side-chains commented May 21, 2024 • edited Loading

qweeah commented May 21, 2024

side-chains commented May 21, 2024

qweeah commented May 21, 2024

qweeah commented May 28, 2024 • edited Loading

sajayantony commented Jun 10, 2024 • edited Loading

github-actions bot commented Aug 10, 2024

github-actions bot commented Sep 9, 2024

side-chains commented May 16, 2024 •

edited

Loading

qweeah commented May 20, 2024 •

edited

Loading

side-chains commented May 21, 2024 •

edited

Loading

qweeah commented May 28, 2024 •

edited

Loading

sajayantony commented Jun 10, 2024 •

edited

Loading