Use Skopeo to pull container images

This commit enables the use of Skopeo to pull container images.
Skopeo uses the OCI schema version 2 to fetch container images.
It lays out the images on disk differently than Docker after a pull.
In order to enable analysis of containers pulled in this way, this PR
introduces the OCIImage class which reflects the expected layout.

In order to deal with the different expected directory structures, we
move a commonly used function in rootfs.py, get_untar_dir(), to the
ImageLayer class, and creates a new property called 'image_layout'.
In this way, container image layouts on disk can be dealt with based
on derived Image classes.

We also introduce the OCIImage class and changes to the
DockerImage class which makes use of the new ImageLayer property
and method. We replace all instances of get_untar_dir() with the
ImageLayer instance's get_untar_dir() method. We connect all
the pieces from the command line option to the image extraction method.
Finally, we add Skopeo to the list of requirements for Tern in the
documentation, Dockerfiles and the development environments.

Lastly, we deal with the different image dictionary layouts
based on the image layout in the html report specifically.

Fixes #948

Signed-off-by: Nisha K <[email protected]>

README.md

@@ -56,7 +56,6 @@ Tern gives you a deeper understanding of your container's bill of materials so y
 
 ![Tern quick demo](/docs/img/tern_demo_fast.gif)
 
-
 # Getting Started<a name="getting-started"/>
 
 ## GitHub Action<a name="github-action"/>
@@ -70,13 +69,16 @@ If you have a Linux OS you will need a distro with a kernel version >= 4.0 (Ubun
 - Python 3.6 or newer (sudo apt-get install python3.6(3.7) or sudo dnf install python36(37))
 - Pip (sudo apt-get install python3-pip).
 - jq (sudo apt-get install jq or sudo dnf install jq)
+- skopeo (See [here](https://github.com/containers/skopeo/blob/main/install.md) for installation instructions or building from source)
 
-Some distro versions have all of these except `attr` and/or `jq` preinstalled but both are common utilities and are available via the package manager.
+Some distro versions have all of these except `attr`, `jq`, and/or `skopeo` preinstalled. `attr` and `jq` are common utilities and are available via the package manager. `skopeo` has only recently been packaged for common Linux distros. If you don't see your distro in the list, your best bet is building from source, which is reasonably straightforward if you have Go installed.
 
-For Docker containers
+For analyzing Dockerfiles and to use the "lock" function
 - Docker CE (Installation instructions can be found here: https://docs.docker.com/engine/installation/#server)
 
-Make sure the docker daemon is running.
+*NOTE:* We do not provide advice on the usage of [Docker Desktop](https://www.docker.com/blog/updating-product-subscriptions/)
+
+Once installed, make sure the docker daemon is running.
 
 Create a python3 virtual environment:
 ```
@@ -103,7 +105,7 @@ $ tern report -o output.txt -i debian:buster
 ```
 
 ## Getting Started with Docker<a name="getting-started-with-docker">
-Docker is the most widely used tool to build and run containers. If you already have Docker installed, you can run Tern by building a container with the Dockerfile provided and the `docker_run.sh` script:
+Docker is the most widely used tool to build and run containers. If you already have Docker installed, you can run Tern by building a container with the Dockerfile provided.
 
 Clone this repository:
 ```
@@ -132,7 +134,13 @@ $ docker build -f ci/Dockerfile -t ternd .
 +ENTRYPOINT ["tern", "-q"]
 ```
 
-Run the script `docker_run.sh`. You may need to use sudo. In the below command `debian` is the docker hub container image name  and `buster` is the tag that identifies the version we are interested in analyzing.
+Run the ternd container image
+
+```
+$ docker run --rm ternd report -i debian:buster
+```
+
+If you are using this container to analyze Dockerfiles and to use the "lock" feature, then you must volume mount the docker socket. We have a convenience script which will do that for you. 
 
 ```
 $ ./docker_run.sh ternd "report -i debian:buster" > output.txt
@@ -143,15 +151,16 @@ To produce a json report run
 $ ./docker_run.sh ternd "report -f json -i debian:buster"
 ```
 
-What the `docker_run.sh` script does is run the built container.
-
 Tern is not distributed as Docker images yet. This is coming soon. Watch the [Project Status](#project-status) for updates.
 
 **WARNING**: If using the `--driver fuse` or `--driver overlay2` storage driver options, then the docker image needs to run as privileged.
+
 ```
-docker run --privileged -v /var/run/docker.sock:/var/run/docker.sock ternd "--driver fuse report -i debian:buster"
+docker run --privileged -v /var/run/docker.sock:/var/run/docker.sock ternd --driver fuse report -i debian:buster
 ```
 
+You can make this change to the `docker_run.sh` script to make it easier.
+
 ## Getting Started with Vagrant<a name="getting-started-with-vagrant">
 Vagrant is a tool to setup an isolated virtual software development environment. If you are using Windows or Mac OSes and want to run Tern from the command line (not in a Docker container) this is the best way to get started as Tern does not run natively in a Mac OS or Windows environment at this time.
 

ci/Dockerfile

@@ -24,6 +24,7 @@ RUN echo "deb http://deb.debian.org/debian bullseye main" > /etc/apt/sources.lis
     fuse3/bullseye \
     git \
     jq \
+    skopeo \
     && rm -rf /var/lib/apt/lists/*
 
 COPY --from=builder /install /usr/local

ci/test_files_touched.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 #
-# Copyright (c) 2019-2020 VMware, Inc. All Rights Reserved.
+# Copyright (c) 2019-2021 VMware, Inc. All Rights Reserved.
 # SPDX-License-Identifier: BSD-2-Clause
 
 from git import Repo
@@ -48,8 +48,12 @@
     # tern/classes
     re.compile('tern/classes/command.py'):
     ['python tests/test_class_command.py'],
+    re.compile('tern/classes/oci_image.py'):
+    ['tern report -i photon:3.0',
+     'python tests/test_class_oci_image.py'],
     re.compile('tern/classes/docker_image.py'):
-    ['tern report -i photon:3.0'],
+    ['tern report -d samples/alpine_python/Dockerfile',
+     'python tests/test_class_docker_image.py'],
     re.compile('tern/classes/file_data.py'):
     ['python tests/test_class_file_data.py'],
     re.compile('tern/classes/image.py'):
@@ -121,6 +125,8 @@
         ['python tests/test_analyze_default_dockerfile_parse.py'],
     re.compile('tests/test_class_command.py'):
         ['python tests/test_class_command.py'],
+    re.compile('tests/test_class_oci_image.py'):
+        ['python tests/test_class_oci_image.py'],
     re.compile('tests/test_class_docker_image.py'):
         ['python tests/test_class_docker_image.py',
          'tern report -w photon.tar'],

docker/Dockerfile

@@ -23,6 +23,7 @@ RUN echo "deb http://deb.debian.org/debian bullseye main" > /etc/apt/sources.lis
     fuse3/bullseye \
     git \
     jq \
+    skopeo \
     && rm -rf /var/lib/apt/lists/*
 
 COPY --from=builder /install /usr/local

tern/__main__.py

@@ -70,8 +70,8 @@ def check_image_input(options):
             logger.critical(errors.incorrect_raw_option)
             sys.exit(1)
     # Check if the image string has the right format
-    if options.docker_image:
-        if not check_image_string(options.docker_image):
+    if options.image:
+        if not check_image_string(options.image):
             logger.critical(errors.incorrect_image_string_format)
             sys.exit(1)
 
@@ -108,9 +108,9 @@ def do_main(args):
                 drun.execute_dockerfile(args)
             else:
                 logger.critical("Currently --layer/-y can only be used with"
-                                " --docker-image/-i")
+                                " --image/-i")
                 sys.exit(1)
-        elif args.docker_image or args.raw_image:
+        elif args.image or args.raw_image:
             check_image_input(args)
             # If the checks are OK, execute for docker image
             crun.execute_image(args)
@@ -167,15 +167,13 @@ def main():
     parser_report.add_argument('-d', '--dockerfile', type=check_file_existence,
                                help="Dockerfile used to build the Docker"
                                " image")
-    parser_report.add_argument('-i', '--docker-image',
-                               help="Docker image that exists locally -"
-                               " image:tag"
-                               " The option can be used to pull docker"
-                               " images by digest as well -"
-                               " <repo>@<digest-type>:<digest>")
+    parser_report.add_argument('-i', '--image',
+                               help="A container image referred either by "
+                               " repo:tag or repo@digest-type:digest")
     parser_report.add_argument('-w', '--raw-image', metavar='FILE',
                                help="Raw container image that exists locally "
-                               "in the form of a tar archive.")
+                               "in the form of a tar archive. Only the output"
+                               "of 'docker save' is supported")
     parser_report.add_argument('-y', '--layer', metavar='LAYER_NUMBER',
                                const=1, action='store',
                                dest='load_until_layer',

tern/analyze/common.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 #
-# Copyright (c) 2017-2020 VMware, Inc. All Rights Reserved.
+# Copyright (c) 2017-2021 VMware, Inc. All Rights Reserved.
 # SPDX-License-Identifier: BSD-2-Clause
 
 '''
@@ -18,7 +18,6 @@
 from tern.utils import cache
 from tern.utils import constants
 from tern.utils import general
-from tern.utils import rootfs
 from debian_inspector import debcon
 from debian_inspector import copyright as debut_copyright
 
@@ -152,7 +151,7 @@ def save_to_cache(image):
 
 def is_empty_layer(layer):
     '''Return True if the given image layer is empty'''
-    cwd = rootfs.get_untar_dir(layer.tar_file)
+    cwd = layer.get_untar_dir()
     if len(os.listdir(cwd)) == 0:
         return True
     return False

tern/analyze/default/container/image.py

@@ -13,6 +13,7 @@
 
 from tern.classes.notice import Notice
 from tern.classes.docker_image import DockerImage
+from tern.classes.oci_image import OCIImage
 from tern.utils import constants
 from tern.analyze import passthrough
 from tern.analyze.default.container import single_layer
@@ -23,24 +24,29 @@
 logger = logging.getLogger(constants.logger_name)
 
 
-def load_full_image(image_tag_string, load_until_layer=0):
-    '''Create image object from image name and tag and return the object.
-    Loads only as many layers as needed.'''
-    test_image = DockerImage(image_tag_string)
+def load_full_image(image_tag_string, image_type='oci', load_until_layer=0):
+    """Create image object from image name and tag and return the object.
+    The kind of image object is created based on the image_type.
+    image_type = oci OR docker
+    Loads only as many layers as needed."""
+    if image_type == 'oci':
+        image = OCIImage(image_tag_string)
+    elif image_type == 'docker':
+        image = DockerImage(image_tag_string)
     failure_origin = formats.image_load_failure.format(
-        testimage=test_image.repotag)
+        testimage=image.repotag)
     try:
-        test_image.load_image(load_until_layer)
+        image.load_image(load_until_layer)
     except (NameError,
             subprocess.CalledProcessError,
             IOError,
             docker.errors.APIError,
             ValueError,
             EOFError) as error:
         logger.warning('Error in loading image: %s', str(error))
-        test_image.origins.add_notice_to_origins(
+        image.origins.add_notice_to_origins(
             failure_origin, Notice(str(error), 'error'))
-    return test_image
+    return image
 
 
 def default_analyze(image_obj, options):

tern/analyze/default/container/multi_layer.py

@@ -30,18 +30,18 @@
 def mount_overlay_fs(image_obj, top_layer, driver):
     '''Given the image object and the top most layer, mount all the layers
     until the top layer using overlayfs'''
-    tar_layers = []
+    layer_paths = []
     for index in range(0, top_layer + 1):
-        tar_layers.append(image_obj.layers[index].tar_file)
-    target = rootfs.mount_diff_layers(tar_layers, driver)
+        layer_paths.append(image_obj.layers[index].get_untar_dir())
+    target = rootfs.mount_diff_layers(layer_paths, driver)
     return target
 
 
 def apply_layers(image_obj, top_layer):
     """Apply image diff layers without using a kernel snapshot driver"""
     # All merging happens in the merge directory
     target = os.path.join(rootfs.get_working_dir(), constants.mergedir)
-    layer_dir = rootfs.get_untar_dir(image_obj.layers[top_layer].tar_file)
+    layer_dir = image_obj.layers[top_layer].get_untar_dir()
     layer_contents = layer_dir + '/*'
     # Account for whiteout files
     for fd in image_obj.layers[top_layer].files:

tern/analyze/default/container/run.py

@@ -14,7 +14,7 @@
 from tern.report import report
 from tern.report import formats
 from tern import prep
-from tern.load import docker_api
+from tern.load import skopeo
 from tern.analyze import common
 from tern.analyze.default.container import image as cimage
 
@@ -27,28 +27,20 @@ def extract_image(args):
     """The image can either be downloaded from a container registry or provided
     as an image tarball. Extract the image into a working directory accordingly
     Return an image name and tag and an image digest if it exists"""
-    if args.docker_image:
-        # extract the docker image
-        image_attrs = docker_api.dump_docker_image(args.docker_image)
-        if image_attrs:
-            # repo name and digest is preferred, but if that doesn't exist
-            # the repo name and tag will do. If neither exist use repo Id.
-            if image_attrs['Id']:
-                image_string = image_attrs['Id']
-            if image_attrs['RepoTags']:
-                image_string = image_attrs['RepoTags'][0]
-            if image_attrs['RepoDigests']:
-                image_string = image_attrs['RepoDigests'][0]
-            return image_string
-        logger.critical("Cannot extract Docker image")
+    if args.image:
+        # download the image
+        result = skopeo.pull_image(args.image)
+        if result:
+            return 'oci', args.image
+        logger.critical("Cannot download Container image: \"%s\"", args.image)
     if args.raw_image:
         # for now we assume that the raw image tarball is always
         # the product of "docker save", hence it will be in
         # the docker style layout
         if rootfs.extract_tarfile(args.raw_image, rootfs.get_working_dir()):
-            return args.raw_image
-        logger.critical("Cannot extract raw image")
-    return None
+            return 'docker', args.raw_image
+        logger.critical("Cannot extract raw Docker image")
+    return None, None
 
 
 def setup(image_obj):
@@ -72,11 +64,11 @@ def teardown(image_obj):
 def execute_image(args):
     """Execution path for container images"""
     logger.debug('Starting analysis...')
-    image_string = extract_image(args)
+    image_type, image_string = extract_image(args)
     # If the image has been extracted, load the metadata
-    if image_string:
+    if image_type and image_string:
         full_image = cimage.load_full_image(
-            image_string, args.load_until_layer)
+            image_string, image_type, args.load_until_layer)
         # check if the image was loaded successfully
         if full_image.origins.is_empty():
             # Add an image origin here

tern/analyze/default/container/single_layer.py

@@ -62,7 +62,7 @@ def find_os_release(host_path):
 def get_os_release(base_layer):
     """Assuming that the layer tarball is untarred are ready to be inspected,
     get the OS information from the os-release file"""
-    return find_os_release(rootfs.get_untar_dir(base_layer.tar_file))
+    return find_os_release(base_layer.get_untar_dir())
 
 
 def get_os_style(image_layer, binary):

tern/analyze/default/debug/run.py

@@ -26,10 +26,10 @@
 from tern.analyze.default.container import multi_layer
 
 
-def check_image_obj(image_string):
+def check_image_obj(image_string, image_type):
     """Return the image object and if it was loaded successfully"""
     if image_string:
-        full_image = cimage.load_full_image(image_string)
+        full_image = cimage.load_full_image(image_string, image_type)
         if full_image.origins.is_empty():
             return full_image, True
         print("Something went wrong in loading the image")
@@ -190,9 +190,9 @@ def recover():
 
 def execute_debug(args):
     """Debug container images"""
-    if args.docker_image:
-        image_string = run.extract_image(args)
-        full_image, success = check_image_obj(image_string)
+    if args.image:
+        image_type, image_string = run.extract_image(args)
+        full_image, success = check_image_obj(image_string, image_type)
         if success:
             if args.keys:
                 # we have an image to mount and some scripts to invoke

tern/analyze/default/default_common.py

@@ -44,7 +44,7 @@ def find_shell(fspath):
 def get_shell(layer):
     '''Find the shell if any on the layer filesystem. Assume that the layer
     has already been unpacked. If there is no shell, return an empty string'''
-    cwd = rootfs.get_untar_dir(layer.tar_file)
+    cwd = layer.get_untar_dir()
     return find_shell(cwd)
 
 
@@ -54,7 +54,7 @@ def get_base_bin(first_layer):
     image and looking for known binaries there. Assume that the layer has
     already been unpacked with the filesystem'''
     binary = ''
-    cwd = rootfs.get_untar_dir(first_layer.tar_file)
+    cwd = first_layer.get_untar_dir()
     for key, value in command_lib.command_lib['base'].items():
         for path in value['path']:
             if os.path.exists(os.path.join(cwd, path)):

tern/analyze/default/dockerfile/run.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 #
-# Copyright (c) 2019-2020 VMware, Inc. All Rights Reserved.
+# Copyright (c) 2019-2021 VMware, Inc. All Rights Reserved.
 # SPDX-License-Identifier: BSD-2-Clause
 
 """
@@ -123,7 +123,7 @@ def full_image_analysis(dfile, options):
     """This subroutine is executed when a Dockerfile is successfully built"""
     image_list = []
     # attempt to load the built image metadata
-    full_image = cimage.load_full_image(dfile)
+    full_image = cimage.load_full_image(dfile, image_type='docker')
     if full_image.origins.is_empty():
         # Add an image origin here
         full_image.origins.add_notice_origin(