Enable inventory of Distroless images

[nishakm]

Describe the Feature
Distroless provides runtime container images. It doesn't have a shell and therefore not as straightforward to enable inventorying via a binary. However, there is a manifest within the rootfs that can read, so inventorying can still be done.

Use Cases
See ossf/wg-supply-chain-integrity#20
We also have difficulty running file level static analysis to get an SBoM for this kind of rootfs, so a deterministic method would be very useful.

Implementation Changes
See comment: ossf/wg-supply-chain-integrity#20 (comment)

The "host" method invokes a host command or script on the working directory without the unshare -pf which you would get from the invoke_in_chroot.

[JamieMagee]

From looking at the referenced comment, I think a sample script would be something like:

files=$(ls var/lib/dpkg/status.d)
for f in $files; do grep -oP "^Package: \K.*$" var/lib/dpkg/status.d/$f; done

Running this in an unpacked layer of gcr.io/distroless/base-debian10 yields

libc6
libssl1.1
openssl

and the version equivalent

for f in $files; do grep -oP "^Version: \K.*$" var/lib/dpkg/status.d/$f; done

yields

2.28-10
1.1.1d-0+deb10u5
1.1.1d-0+deb10u5

etc. for the other commands.

[JamieMagee]

@nishakm I've been looking into implementing this. You mentioned about using the host method to invoke commands, and the documentation also says this:

tern/tern/analyze/default/command_lib/base.yml

Lines 12 to 17 in 11fc8a3

	# names: <a list of package names>
	# invoke: (if this is a script to invoke include this)
	# 1:
	# container: <list of commands> (if the environment is the container)
	# 2:
	# host: <list of commands> (if the environment is the host)

However, the implementation only appears to be there for container, and host is missing?

tern/tern/report/content.py

Lines 78 to 87 in 11fc8a3

	if 'invoke' in info_dict[info]:
	report = report + info + ':\n'
	for step in range(1, len(info_dict[info]['invoke'].keys()) + 1):
	if 'container' in info_dict[info]['invoke'][step]:
	report = report + formats.invoke_in_container
	for snippet in info_dict[info]['invoke'][step]['container']:
	report = report + '\t' + snippet + '\n'
	else:
	for value in info_dict[info]:
	report = report + ' ' + value

[nishakm]

@JamieMagee What I meant was that host is meant to be one of ways tern can run scripts (and something that I had implemented a long long time ago and since removed from the code base) but the execution path is not implemented yet.

The place to implement this is in https://github.com/tern-tools/tern/blob/main/tern/analyze/default/collect.py#L22
and in https://github.com/tern-tools/tern/blob/main/tern/analyze/default/collect.py#L60

Here we also need to add a new function invoke_on_host which just execs the above scripts wherever tern is currently running. Now this is going to be really tricky on Windows and Mac hosts because we are now relying on the host's shell rather than the container's shell. I'm open to ideas on how to tackle this :)

[JamieMagee]

@nishakm I think we're on the same page. It wasn't clear to me whether invoke_on_host had yet to be implemented, had been removed in a refactor, or I was missing something in the codebase. Regardless, it needs to be implemented now.

As for the issue of Windows and Mac hosts, I would defer that question for now and assume that we have a Linux host. Let's get something working first, then worry about crossplatform support afterwards.

[JamieMagee]

It's hacked together right now, but I've proven the concept:

Docker image: gcr.io/distroless/base@sha256:dd04859fc2fd78672388a1f60a0ddc935bf7c365279c14f7533606ffc9d446be:
        Layer 1:
                warning: No known shell found in image. Cannot invoke commands

                warning: Unable to find a known package manager. Cannot list packages.

                info: Found 'Distroless' in /etc/os-release.
        File licenses found in Layer:  None
        Packages found in Layer:  base-files-9.9+deb9u13, netbase-5.4, tzdata-2021a-0+deb9u1
        Licenses found in Layer:  None

[nishakm]

@rnjudge I think we may remove the distroless issue from GSoC? What do you think we can substitute it with?

[nishakm]

It's hacked together right now, but I've proven the concept:

Docker image: gcr.io/distroless/base@sha256:dd04859fc2fd78672388a1f60a0ddc935bf7c365279c14f7533606ffc9d446be:
        Layer 1:
                warning: No known shell found in image. Cannot invoke commands

                warning: Unable to find a known package manager. Cannot list packages.

                info: Found 'Distroless' in /etc/os-release.
        File licenses found in Layer:  None
        Packages found in Layer:  base-files-9.9+deb9u13, netbase-5.4, tzdata-2021a-0+deb9u1
        Licenses found in Layer:  None

@JamieMagee Feel free to submit a PR with what you have. I can either review it with changes or work on top of it. Whatever works for you.

[rnjudge]

@rnjudge I think we may remove the distroless issue from GSoC? What do you think we can substitute it with?

@nishakm #757?

[nishakm]

@rnjudge I think we may remove the distroless issue from GSoC? What do you think we can substitute it with?

@nishakm #757?

https://github.com/tern-tools/tern/wiki/Google-Summer-of-Code-2021-Ideas#enable-inventory-of-distroless-container-images-beginnerintermediate