[FEATURE] Ability to run soci create command in CodeBuild

[tmokmss]

Description

Hi, when I try to run soci create command in a CodeBuild environment, I get the following error:

[Container] 2023/08/01 11:56:35 Running command sudo soci create $REGISTRY/repository:tag
soci: failed to dial "/run/containerd/containerd.sock": context deadline exceeded: connection error: desc = "transport: error while dialing: dial unix:///run/containerd/containerd.sock: timeout"

I guess it's because containerd is not available in the environment. However, as CodeBuild job itself runs on a container environment, I think it isn't possible to install containerd and run its service inside the container. Because of this limitation, it's difficult to build and push a SOCI index from CodeBuild, which is a common use case in CI/CD context.

Can't we remove this dependency on containerd service from the soci CLI? In this project, SOCI index is built on a Lambda environment, which does not have containerd service running. I prefer this approach with less prerequisites on the build environment.

Describe the solution you'd like

cfn-ecr-aws-soci-index-builder seems to have already resolved this problem. I think we can follow this approach, but I'm not aware of any downside of it.

Describe any alternative solutions/features you've considered

No response

Any additional context or information about the feature request

I created a prototype CLI to impement this idea (most of the code is just copied from the above project 😅 ). I confirmed it runs on a CodeBuild job successfully. I'd like to use soci create command in the same way :)

https://github.com/tmokmss/soci-wrapper/

[Kern--]

We could consider adding a flag to allow you to control where to find the image content (e.g. --store containerd or --store registry - these names are not very good, but just to demonstrate).

I would expect that in many cases, the image that you want to build a SOCI index for is also being built by the same CodeBuild pipeline, right? There's some waste there where doing this would 1) create an image, 2) push the imaged to the registry, 3) pull the image from the registry with SOCI, 4) create and push a SOCI index. If I remember correctly, docker stores uncompressed tarballs locally and only compresses them on push so we couldn't just read directly from the docker storage. [citation needed]

Possibly another approach would be to try to get SOCI into buildkit so you could build your SOCI index with docker buildx. I'm not sure how widely used buildkit is in the wild though, so this might only be a niche solution.

[tmokmss]

@Kern-- I see, thanks. Yeah it's desired to have an option to use soci-snapshotter on environments like CodeBuild even though it's inefficient. I think buildkit is also preferable as it's often available on CI environments such as CodeBuild or GHA.

[tmokmss]

fyi I just published a CDK construct to build and publish SOCI indices during CloudFomation deployment.

https://constructs.dev/packages/deploy-time-build/v/0.3.5?lang=typescript#build-soci-index-for-a-container-image

[Kern--]

We're going to take a look at this, but we want to answer the following questions:

1. How many people are using/want SOCI in codebuild?
2. What would it look like to move the index builder code into the SOCI codebase
3. Is the index builder code the right way to solve this problem?

[tmokmss]

@Kern-- Thanks for considering this! Regarding question 1, I don't have any concrete metrics, but at least here's my observation (correct me if I'm wrong!)

As far as I know, there are currently two official ways to build SOCI index; 1. use soci-snapshotter CLI 2. use cfn-ecr-aws-soci-index-builder.

soci-snapshotter currently only runs on a Linux machine with containerd installed, e.g. an EC2 instance or a user's local machine, but I guess they can only be used for testing purposes. If it cannot run on a CI environment like CodeBuild, then how can we use SOCI indexes for a production environment?

cfn-ecr-aws-soci-index-builder also has some possible problems (it should work in most cases though). Since it runs on Lambda, it has some limitations on image size (< 6GB?), execution time (15min), etc. Also, because it runs asynchronously after an image is pushed, sometimes the ECS service starts running tasks before a SOCI index is actually pushed.

You might want to consider that, because the SOCI index won't be there yet when you deploy immediately after push
(quote from a customer)

So I think there's certainly some missing pieces that should be filled in. Hope this helps, thanks!

[sparr]

If it cannot run on a CI environment like CodeBuild, then how can we use SOCI indexes for a production environment?

It will work in server-ful and non-ephemeral CI environments, where the necessary services (containerd, soci) can be available. Although I don't necessarily recommend using Github Actions for production CI, their runners are capable of the necessary steps.

[tmokmss]

I see, thanks. ...but we love serverless and that's why we use Fargate :)

[ollypom]

Just an FYI, I do have a sample of generating an Index in a CodeBuild environment here. https://github.com/aws-samples/aws-fargate-seekable-oci-toolbox/tree/main/soci-codepipeline.

As per @tmokmss comment:

I guess it's because containerd is not available in the environment. However, as CodeBuild job itself runs on a container environment, I think it isn't possible to install containerd and run its service inside the container. Because of this limitation, it's difficult to build and push a SOCI index from CodeBuild, which is a common use case in CI/CD context.

You can actually run a separate containerd in the codebuild environment, and use that to generate the index.

[dil-skilana]

I am facing the same error while trying to create a SOCI index soci create in aws codebuild. Can anyone suggest an alternate way to make it work in the aws codepipeline build stage.

soci: failed to dial "/run/containerd/containerd.sock": context deadline exceeded: connection error: desc = "transport: error while dialing: dial unix:///run/containerd/containerd.sock: timeout"

[pendo324]

I am facing the same error while trying to create a SOCI index soci create in aws codebuild. Can anyone suggest an alternate way to make it work in the aws codepipeline build stage.

soci: failed to dial "/run/containerd/containerd.sock": context deadline exceeded: connection error: desc = "transport: error while dialing: dial unix:///run/containerd/containerd.sock: timeout"

It looks like you haven't started containerd inside of your container. Have you taken a look at Olly's comment? #760 (comment)

Another option may be to install Finch. It will come preinstalled on macOS codebuild runners (usage guide), and can be installed on Amazon Linux by following these instructions.

This will allow you to create SOCI indexes like so:

AWS_ACCOUNT_ID=111222333444
AWS_REGION=eu-west-1

finch push --snapshotter soci \
    $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/myimage:latest