Use Skopeo to pull container images #1090
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rnjudge
reviewed
Dec 14, 2021
rnjudge
reviewed
Dec 14, 2021
rnjudge
reviewed
Dec 15, 2021
rnjudge
reviewed
Dec 15, 2021
rnjudge
reviewed
Dec 15, 2021
|
I tested this with Dockerfiles, multi-layer images and problematic past images/scenarios and everything looked as expected when running natively on Linux. I did get some weird race-y warning messages at times but could never reliably reproduce them. Once the nits/suggestions are resolved this is good to go :) |
This is work towards tern-tools#948 OCI images on disk may be represented in different ways. As such the layer tarballs may exist at the root of the working directory rather than in their own path. This change allows alternative directory paths to be used based on the image layout. - Added a new property: image_layout to the ImageLayer class. The image_layout property will contain either "docker" or "oci" with the default being "oci" - Moved get_untar_dir from rootfs.py to be a method in the ImageLayer class. get_untar_dir will now return a directory path to the expected path where the layer's contents are untarred based on the image layout. Removed an unnecessary function in rootfs.py as a result. - Modified the mount_diff_layers to accept untarred directory paths instead of tar files. - Modified the function mount_overlay_fs in multi_layer.py to collect the directory paths instead of the tar file to pass to the mount_diff_layers function. - Added a new get_untar_dir test to the ImageLayer test suite. Signed-off-by: Nisha K <[email protected]>
nishakm
force-pushed
the
948-skopeo-copy
branch
from
58b98bb to
7f2fdab
Compare
rnjudge
reviewed
Dec 15, 2021
added 5 commits
December 15, 2021 12:07
This is work towards tern-tools#948 This change adds a new class - OCIImage which represents the OCI image schema version 2 available on disk when downloaded by skopeo. This change also adds a skopeo.py script to the list of loading methods. Skopeo.py adds functions to check if the skopeo binary exists on disk and a function to pull images using skopeo. A test suite for the OCIImage class is also added. This test suit is similar to the test_class_docker_image.py test suit. Additionally, we have changed the DockerImage class and associated tests to set the layer's image_layout property to "docker". We've switched the test for changes to the oci class to use a container image and the changes to the docker class to use the dockerfile, as we need the docker daemon to test Dockerfile builds. Signed-off-by: Mukul Taneja <[email protected]> Signed-off-by: Nisha K <[email protected]>
This is work towards tern-tools#948 This commit replaces all instances of the old rootfs get_untar_dir function with the ImageLayer method get_untar_dir. We also remove an unused function in passthrough.py that uses layer tar files. Signed-off-by: Nisha K <[email protected]>
This is work towards tern-tools#948 This commit makes changes to hook up the CLI option for container image to use the OCIImage class rather than the DockerImage class. The argument for the -i option is now --image. - If the raw_image option is given, we assume that this is the output of "docker save" rather than some other tool. - In the load_full_image function, we provide an "image_type" string which should either be "oci" or "docker", "oci" being the default. Based on this string, we either return an instance of OCIImage or DockerImage. - For the Dockerfile or Lock features, we explicitly create DockerImage objects. Signed-off-by: Nisha K <[email protected]>
This is work towards tern-tools#948 Now that skopeo has become the de-facto tool to download container images, we will switch our instructions to install skopeo and use it in our build and development environments. At this time, skopeo is available on Ubuntu 20.04 through the Kubic project, hence there may need to be some additional changes in the future as the upstream package creators improve. Signed-off-by: Nisha K <[email protected]>
This is work towards tern-tools#948 The html formatting of lists allows for an expansion UI under the title of the layer object. The title is the layer tarball's digest. In the Docker image layout, this can be directly taken from the "tar_file" key. But in the OCI layout, this value is taken from the "digest" key. This commit allows for that difference and defaults to the layer's index if neither one of those keys exist. Signed-off-by: Nisha K <[email protected]>
nishakm
force-pushed
the
948-skopeo-copy
branch
from
7f2fdab to
3bdbd08
Compare
rnjudge
approved these changes
Dec 15, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR enables the use of Skopeo to pull container images.
Skopeo uses the OCI schema version 2 to fetch container images.
It lays out the images on disk differently than Docker after a pull.
In order to enable analysis of containers pulled in this way, this PR
introduces the OCIImage class which reflects the expected layout.
In order to deal with the different expected directory structures, commit 1
moves a commonly used function in rootfs.py, get_untar_dir, into the
ImageLayer class, and creates a new property called 'image_layout'.
In this way, container image layouts on disk can be dealt with based
on derived Image classes.
Commit 2 introduces the OCIImage class and changes to the
DockerImage class which makes use of the new ImageLayer property
and method. Commit 3 replaces all instances of get_untar_dir with the
ImageLayer instance's get_untar_dir method. Commit 4 connects all
the pieces from the command line option to the image extraction method.
Finally, commit 5 adds Skopeo to the list of requirements for Tern in the
documentation, Dockerfiles and the development environments.
Commit 6 deals with the different image dictionary layouts based on the
image layout in the html report specifically.
Note that the change to the Dockerfile that installs tern using pip has not
changed as this changeset is not included in any releases yet.
Fixes #948
Signed-off-by: Nisha K [email protected]