Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(k8s)!: support k8s multi container #7444

Merged
merged 4 commits into from
Oct 29, 2024

Conversation

smtan-gl
Copy link
Contributor

@smtan-gl smtan-gl commented Sep 5, 2024

Description

When using Trivy k8s, the current report metadata field for workloads with multiple containers only includes metadata from one image.
This update changes the metadata field to an array, allowing it to capture metadata from all the scanned images within the workload.

This is a draft PR to gather feedback on whether this approach is acceptable, and I will update the tests accordingly thereafter.

Report before change
{
  "ClusterName": "",
  "Findings": [
    {
      "Namespace": "default",
      "Kind": "Pod",
      "Name": "nginx-fluentd-pod",
      "Metadata": {
        "OS": {
          "Family": "debian",
          "Name": "12.6"
        },
          "RepoTags": [
            "nginx:latest"
          ],
          "DiffIDs": []
      },
      "Results": [
        {
          "Target": "nginx:latest (debian 12.6)",
          "Class": "os-pkgs",
          "Type": "debian",
          "Packages": [],
          "Vulnerabilities": []
        },
        {
          "Target": "fluent/fluentd:v1.17-armhf-debian (debian 12.6)",
          "Class": "os-pkgs",
          "Type": "debian",
          "Packages": [],
          "Vulnerabilities": []
        },
        {
          "Target": "Ruby",
          "Class": "lang-pkgs",
          "Type": "gemspec",
          "Packages": [],
          "Vulnerabilities": []
        }
      ]
    }
  ]
}
Report after change
{
  "ClusterName": "",
  "Findings": [
    {
      "Namespace": "default",
      "Kind": "Pod",
      "Name": "nginx-fluentd-pod",
      "Metadata": [
        {
          "OS": {
            "Family": "debian",
            "Name": "12.6"
          },
          "RepoTags": [
            "nginx:latest"
          ],
          "DiffIDs": []
        },
        {
          "OS": {
            "Family": "debian",
            "Name": "12.6"
          },
          "RepoTags": [
            "fluent/fluentd:v1.17-armhf-debian"
          ],
          "DiffIDs": []
        }
      ],
      "Results": [
        {
          "Target": "nginx:latest (debian 12.6)",
          "Class": "os-pkgs",
          "Type": "debian",
          "Packages": [],
          "Vulnerabilities": []
        },
        {
          "Target": "fluent/fluentd:v1.17-armhf-debian (debian 12.6)",
          "Class": "os-pkgs",
          "Type": "debian",
          "Packages": [],
          "Vulnerabilities": []
        },
        {
          "Target": "Ruby",
          "Class": "lang-pkgs",
          "Type": "gemspec",
          "Packages": [],
          "Vulnerabilities": []
        }
      ]
    }
  ]
}

Related issues

Checklist

  • I've read the guidelines for contributing to this repository.
  • I've followed the conventions in the PR title.
  • I've added tests that prove my fix is effective or that my feature works.
  • I've updated the documentation with the relevant information (if needed).
  • I've added usage information (if the PR introduces new options)
  • I've included a "before" and "after" example to the description (if the PR is a user interface change).

@smtan-gl smtan-gl requested a review from afdesk as a code owner September 5, 2024 03:10
@CLAassistant
Copy link

CLAassistant commented Sep 5, 2024

CLA assistant check
All committers have signed the CLA.

@@ -65,10 +65,26 @@ type Resource struct {
Report types.Report `json:"-"`
}

type ConsolidatedResource struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we update Resource? I think it's a bit hard to keep 2 similar structs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also there are no tests for this case.

Copy link
Contributor Author

@smtan-gl smtan-gl Sep 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback, I'm not sure why I assumed the Resource struct was used by other packages in Trivy, but it doesn't seem to be the case. I'll work on updating the Resource struct accordingly.

also there are no tests for this case.

Regarding the tests, I wanted to get feedback on the approach before proceeding. I'll focus on adding the tests next.

@afdesk
Copy link
Contributor

afdesk commented Sep 5, 2024

Hi @smtan-gl thanks for the contribution! it's really nice and important.

I left some comments about this PR.

also I took a look at #5889
it seems we have to add more tests for these cases...

@smtan-gl
Copy link
Contributor Author

smtan-gl commented Sep 6, 2024

Hi @smtan-gl thanks for the contribution! it's really nice and important.

I left some comments about this PR.

also I took a look at #5889 it seems we have to add more tests for these cases...

@afdesk any chance you might have context on this question?

@afdesk
Copy link
Contributor

afdesk commented Sep 10, 2024

@afdesk any chance you might have context on this question?

If I understand correctly there is an option to create an image with several tags, so it should be a slice.
https://stackoverflow.com/questions/31963525/is-it-possible-for-image-to-have-multiple-tags

@smtan-gl smtan-gl force-pushed the fix_k8s_multi_container branch from 1571d0d to c99c624 Compare September 12, 2024 09:33
@smtan-gl smtan-gl changed the title fix(k8s): support k8s multi container(DRAFT) fix(k8s): support k8s multi container Sep 12, 2024
@smtan-gl
Copy link
Contributor Author

@afdesk I've updated the PR, could you have another look please 😄

@afdesk
Copy link
Contributor

afdesk commented Sep 16, 2024

@afdesk I've updated the PR, could you have another look please 😄

sure, thanks! I'll take a look

@afdesk
Copy link
Contributor

afdesk commented Sep 17, 2024

@smtan-gl could you sign CLA for a while? thanks!

@smtan-gl
Copy link
Contributor Author

@smtan-gl could you sign CLA for a while? thanks!

@afdesk, I've just signed it. Thank you.

Copy link
Contributor

@afdesk afdesk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribute! I left a few comments

pkg/k8s/report/report_test.go Outdated Show resolved Hide resolved
pkg/k8s/report/report_test.go Outdated Show resolved Hide resolved
@afdesk afdesk changed the title fix(k8s): support k8s multi container fix(k8s)!: support k8s multi container Sep 18, 2024
@afdesk
Copy link
Contributor

afdesk commented Sep 18, 2024

it's a BREAKING change, so we have to create a new discussion about it.
like here: #7010
we'll do it after the PR is merged. just to remember about it

@afdesk
Copy link
Contributor

afdesk commented Sep 18, 2024

@smtan-gl also, I'm not sure that this PR fixes #5889...

@afdesk
Copy link
Contributor

afdesk commented Sep 18, 2024

@smtan-gl please, feel free to correct me if I miss something.
I've run an empty minikube. and then I create a pod with 2 images in my test namespace:

apiVersion: v1
kind: Pod
metadata:
  name: my-multiimage-pod
spec:
  containers:
  - name: my-image1-nginx
    image: nginx
  - name: my-image2-alpine-sleep
    image: alpine:3.17.1
    command: [ "/bin/sh", "-c", "--" ]
    args: [ "while true; do sleep 30; done;"]
$ kubectl create namespace k8s-test
$ kubectl create -f multiimagepod.yaml -n k8s

and try to scan this cluster with the update:

$ ./tr k8s --report all -f json -o res2.json --include-namespaces k8s-test --timeout 30m --debug --include-kinds Pod --scanners vuln

Metadata still contains only one record:
изображение
изображение

@smtan-gl
Copy link
Contributor Author

@afdesk, the bug #5889 occurs only with the --report summary configuration.

In your example, you used --report all which does not consolidate the report. As such, there are 2 reports in the Resources field, 1 for each of the scanned images containing only 1 metadata.

When I run the command using --report summary:

$ ./tr k8s --report summary -f json -o res2.json --include-namespaces k8s-test --timeout 30m --debug --include-kinds Pod --scanners vuln

It consolidates the reports into 1 consolidated report in the Findings field, and you can see that there are 2 metadata for the 2 images.

Screenshot 2024-09-19 at 2 02 05 PM

@smtan-gl smtan-gl requested a review from afdesk September 19, 2024 06:35
Copy link
Contributor

@afdesk afdesk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@afdesk
Copy link
Contributor

afdesk commented Sep 19, 2024

@afdesk, the bug #5889 occurs only with the --report summary configuration.

exactly! thanks for clarify!

@smtan-gl
Copy link
Contributor Author

Thanks for reviewing the PR @afdesk 😄 May I know what's the next steps needed to have this PR merged?

@afdesk
Copy link
Contributor

afdesk commented Sep 20, 2024

Thanks for reviewing the PR @afdesk 😄 May I know what's the next steps needed to have this PR merged?

@smtan-gl thanks for your contribution again.
It's a breaking change so let's wait feedback from another maintainers and community

@smtan-gl
Copy link
Contributor Author

Thanks for reviewing the PR @afdesk 😄 May I know what's the next steps needed to have this PR merged?

@smtan-gl thanks for your contribution again. It's a breaking change so let's wait feedback from another maintainers and community

Hi @afdesk would you have an estimate on how long we need to wait for feedback?

@afdesk
Copy link
Contributor

afdesk commented Sep 25, 2024

Hi @smtan-gl

there is a concern about this solution: how people associate the result with the metadata if it's N x N?

wdyt about it? thanks!

@smtan-gl
Copy link
Contributor Author

Hi @smtan-gl

there is a concern about this solution: how people associate the result with the metadata if it's N x N?

wdyt about it? thanks!

Thanks for the response @afdesk. I thought we could match the vulnerability's Layer.DiffID with the metadata DiffIDs, but I've realized that the DiffIDs across multiple metadata entries might not be unique. Would it make sense to introduce an ImageID field to vulnerability?

For your convenience, I've attached below a JSON report from a scan with changes from this MR.

multi-image-pod.json

@afdesk
Copy link
Contributor

afdesk commented Sep 30, 2024

@smtan-gl there are a few ideas.

  1. @knqyf263 suggests to have Metadata and Results per image.
a sample json for a new field
  "Images": [
    {
      "Metadata": {
        "OS": {
          "Family": "debian",
          "Name": "12.6"
        },
        "RepoTags": [
          "image-a:latest"
        ]
      },
      "Results": [
        {
          "Target": "image-a:latest(debian 12.6)",
          "Class": "os-pkgs",
          "Type": "debian",
          "Vulnerabilities": []
        }
      ]
    },
    {
      "Metadata": {
        "OS": {
          "Family": "ubuntu",
          "Name": "22.04"
        },
        "RepoTags": [
          "image-b:latest"
        ]
      },
      "Results": [
        {
          "Target": "image-b:latest(ubuntu 22.04)",
          "Class": "os-pkgs",
          "Type": "ubuntu",
          "Vulnerabilities": []
        }
      ]
    }
  ]

Next ideas came from the point summary report basically is for reference and for table output.

  1. disable JSON format for summary reports.
    People can summarize results by themselves. and maybe someone will create a plugin for it.

  2. Pods are the smallest deployable units of computing that you can create and manage in Kubernetes. so If we say Pod is a single object, and we don't distinguish image. and then the current approach is fine.

Wdyt @smtan-gl ?

@smtan-gl
Copy link
Contributor Author

smtan-gl commented Oct 2, 2024

Next ideas came from the point summary report basically is for reference and for table output.

Thanks for the response @afdesk. I agree with your points. However, I assume that there are other users who already rely on summary report in JSON format, so disabling it might affect them. For that reason, I support moving forward with approach 3.

@afdesk
Copy link
Contributor

afdesk commented Oct 14, 2024

@smtan-gl just small update that I didn't forget about your job.
I hope we'll resolve it before next release.
thanks again

@knqyf263 knqyf263 added this pull request to the merge queue Oct 29, 2024
Merged via the queue into aquasecurity:main with commit c434775 Oct 29, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unexpected JSON output when --report summary is set in trivy k8s for multi-container workloads
4 participants