OCI Image Updater use-case

[weeshal]

Library name

Azure.Containers.ContainerRegistry

Please describe the feature.

We would like to use the azure sdk to automate our container deployment process. At a specified cadence, an agent running on an edge machine would pull the latest artifacts from an Azure Container Registry and load that image onto the machine. The registry is not configured for anonymous access - our workflow currently is to utilize a pull-token to authenticate the download. We would ideally retrieve the token password through an Azure KeyVault SecretClient and use it to pull the desired image. Specifically, we would like to call DownloadBlobAsync.

Based on my understanding, the current SDK only supports anonymous access operations, but I would like to inquire about any potential plans to support authentication through a token/password.

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[annelo-msft]

Thanks for your question, @weeshal!

Based on my understanding, the current SDK only supports anonymous access operations

The library does support anonymous access. It also supports authentication via AAD, as described here.

Would this be sufficient for your purposes?

[weeshal]

Hey Anne, yes that was sufficient in terms of authentication. However, wanted to follow up on downloading larger blobs - the single DownloadBlobAsync method times out when downloading a larger artifact (~5GB). Here is the error I am facing:
System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry. (Stream was too long.) (Stream was too long.) (Stream was too long.) (Stream was too long.) ---> System.IO.IOException: Stream was too long. at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.IO.MemoryStream.WriteAsync(ReadOnlyMemory1 buffer, CancellationToken cancellationToken) --- End of stack trace from previous location --- at Azure.Core.Pipeline.ResponseBodyPolicy.CopyToAsync(Stream source, Stream destination, CancellationTokenSource cancellationTokenSource) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) --- End of inner exception stack trace --- at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Containers.ContainerRegistry.ContainerRegistryBlobRestClient.GetBlobAsync(String name, String digest, CancellationToken cancellationToken) at Azure.Containers.ContainerRegistry.Specialized.ContainerRegistryBlobClient.DownloadBlobAsync(String digest, CancellationToken cancellationToken) at AzcRobotAgent.ContainerUpdateWorker.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\vishalgi\Azure-Compute-Robotics\src\dotnet\AzcRobotAgent\ContainerUpdateWorker.cs:line 145 ---> (Inner Exception #1) System.IO.IOException: Stream was too long. at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.IO.MemoryStream.WriteAsync(ReadOnlyMemory1 buffer, CancellationToken cancellationToken) --- End of stack trace from previous location --- at Azure.Core.Pipeline.ResponseBodyPolicy.CopyToAsync(Stream source, Stream destination, CancellationTokenSource cancellationTokenSource) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)<--- ---> (Inner Exception #2) System.IO.IOException: Stream was too long. at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.IO.MemoryStream.WriteAsync(ReadOnlyMemory1 buffer, CancellationToken cancellationToken) --- End of stack trace from previous location --- at Azure.Core.Pipeline.ResponseBodyPolicy.CopyToAsync(Stream source, Stream destination, CancellationTokenSource cancellationTokenSource) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)<--- ---> (Inner Exception #3) System.IO.IOException: Stream was too long. at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.IO.MemoryStream.WriteAsync(ReadOnlyMemory1 buffer, CancellationToken cancellationToken) --- End of stack trace from previous location --- at Azure.Core.Pipeline.ResponseBodyPolicy.CopyToAsync(Stream source, Stream destination, CancellationTokenSource cancellationTokenSource) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)<---`

Any suggestions on this?

[annelo-msft]

Hi @weeshal, I'm glad to hear you were able to authenticate and call DownloadBlob!

This is a known issue with large blobs, and the resolution is being tracked by #32414. The fix will not release in beta 5, but we intend it to come in one of the following betas.

If this is your remaining outstanding issue, may I close this issue and have you follow that one instead?

[weeshal]

Sounds good to me. Thank you!

[weeshal]

While on the subject, I noticed that the digest received from the manifest directly was not the one that referred to the actual image itself. It required drilling into the first (and in our case, only) layer, and using the digest within the OciBlobDescriptor. Not sure if this already tracked, or intended, but it would be great to use the root level digest to download the entire image - It seems the manifest at the root level is of type application/vnd.unknown.config.v1+json instead of application/vnd.oci.image.config.v1+json. See more info here: https://oras.land/cli/3_manifest_config/.

[annelo-msft]

Thanks @weeshal!

Sounds good to me. Thank you!

Great, I'm closing this issue in response to this.

While on the subject, I noticed that the digest received from the manifest directly was not the one that referred to the actual image itself.

Interesting! I'm curious to learn more and help solve this. Would you be willing to file a new issue (so the title reflects the issue) with a repro description?

Many thanks!

[annelo-msft]

@weeshal, I am wondering if this PR will address the issue you've described in this comment.