"--include-kinds pod --disable-node-collector" fails with the "nodes is forbidden" error.

[jeremychoi]

Description

It might be related to #6653, but seems more related to the RBAC configuration in the cluster in my case. That's because, when I run the following command:

$ ./trivy-0.53 k8s --include-namespaces "default" --include-kinds pod --disable-node-collector --scanners=misconfig --skip-images --report summary --exit-code 0

I have no issue when I run as kubeadmin, but I get an error when I run as only a limited privileged account(but I'm given to the access to the namespace):

Fatal error get k8s artifacts error: nodes is forbidden: User "jechoi" cannot list resource "nodes" in API group "" at the cluster scope.

Note that "kubectl get pod -n default" is working as I have access to the namespace. The problem seems to come from the fact I don't have privilege to access the "nodes", but I added "--include-kinds pod" so the command should work.

Just as a side note, with v0.49, the following command ran successfully without any issue.

$ ./trivy-0.49.1 k8s -n "default" pod --scanners=misconfig --report summary

Desired Behavior

The command with "--include-kinds pod" with "--disable-node-collector" runs successfully without the fatal error from the "nodes".

Actual Behavior

I got an error like:

Fatal error get k8s artifacts error: nodes is forbidden: User "jeremychoi" cannot list resource "nodes" in API group "" at the cluster scope.

Reproduction Steps

1. set up a cluster and give a user only access to a namespace (not kubeadmin)
2. run a Trivy k8s scan command against the namespace as the Debug Output (in fact I also tested with v0.52.2 which resulted in the same error)

Target

Kubernetes

Scanner

Misconfiguration

Output Format

Table

Mode

Standalone

Debug Output

$ ./trivy-0.53 k8s --include-namespaces "default" --include-kinds pod --disable-node-collector --scanners=misconfig --skip-images --report summary --exit-code 0 --debug
2024-07-03T23:35:42-04:00       DEBUG   Cache dir       dir="/home/jechoi/.cache/trivy"
2024-07-03T23:35:42-04:00       DEBUG   Parsed severities       severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2024-07-03T23:35:42-04:00       DEBUG   Ignore statuses statuses=[]
...(snipped)..
2024-07-03T23:35:56-04:00       FATAL   Fatal error     get k8s artifacts error: nodes is forbidden: User "jechoi" cannot list resource "nodes" in API group "" at the cluster scope

Operating System

Linux fedora

Version

$ trivy --version
Version: 0.53.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-03-12 00:16:41.533348297 +0000 UTC
  NextUpdate: 2024-03-12 06:16:41.533347896 +0000 UTC
  DownloadedAt: 2024-03-12 03:10:06.875028648 +0000 UTC
Java DB:
  Version: 1
  UpdatedAt: 2024-03-11 00:50:00.808963767 +0000 UTC
  NextUpdate: 2024-03-14 00:50:00.808963647 +0000 UTC
  DownloadedAt: 2024-03-12 03:15:51.948040996 +0000 UTC
Check Bundle:
  Digest: sha256:ef2d9ad4fce0f933b20a662004d7e55bf200987c180e7f2cd531af631f408bb3
  DownloadedAt: 2024-07-03 03:21:40.421543728 +0000 UTC

Checklist

• Run trivy clean --all
• Read the troubleshooting

[afdesk]

Hi @jeremychoi!
#6653 should be fixed in v0.54.1.

could you confirm the issue is still relevant for your case with the latest version?
thanks

[afdesk]

Hi @jeremychoi!
it seems i could reproduce your issue.
I've created a user with limited credentials and try to scan a cluster.
Trivy 0.54.1 doesn't show fatal error anymore, but I'm not sure that it scans correct.

[jeremychoi]

@afdesk Many thanks for looking into this. However, the problem still occurs. Please see the following logs. Note I can get the pod info from the namespace via 'kubectl get pod -n results-collector--runtime-int'. I suspect Trivy seems to check the 'cluster' scope first and fail the scan if there's no proper permission at the cluster scope. What's expected should be, IMHO, just skipping the checks and proceed for scanning for the 'namespace' scope resources. Please note as I mentioned earlier in the comment above, v0.49 is working successfully against the same cluster environment.

$ trivy --version
Version: 0.54.1
Check Bundle:
Digest: sha256:ef2d9ad4fce0f933b20a662004d7e55bf200987c180e7f2cd531af631f408bb3
DownloadedAt: 2024-07-04 23:47:55.927796257 +0000 UTC
[jechoi@fedora trivy]$ trivy kubernetes --include-namespaces "results-collector--runtime-int" --include-kinds pod --disable-node-collector --scanners=misconfig --report all --debug
2024-08-15T01:27:57-04:00 DEBUG Cache dir dir="/home/jechoi/.cache/trivy"
2024-08-15T01:27:57-04:00 DEBUG Parsed severities severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2024-08-15T01:27:57-04:00 DEBUG Ignore statuses statuses=[]
2024-08-15T01:27:59-04:00 ERROR Unable to list resources error="failed listing resources for gvr: apps/v1, Resource=deployments - deployments.apps is forbidden: User "jechoi" cannot list resource "deployments" in API group "apps" at the cluster scope"
2024-08-15T01:27:59-04:00 ERROR Unable to list resources error="failed listing resources for gvr: /v1, Resource=pods - pods is forbidden: User "jechoi" cannot list resource "pods" in API group "" at the cluster scope"
2024-08-15T01:28:00-04:00 ERROR Unable to list resources error="failed listing resources for gvr: apps/v1, Resource=replicasets - replicasets.apps is forbidden: User "jechoi" cannot list resource "replicasets" in API group "apps" at the cluster scope"
...

[afdesk]

@jeremychoi thanks for the details.
I meant there is no fatal error now, but you're right. Trivy doesn't scan correctly througt a limited user.

[jeremychoi]

I'm just wondering if there's any news on this?

[jeremychoi]

or how could I help?

[afdesk]

Hi @jeremychoi! thank you so mush for your offer! it's really nice.
I have some idea and will update you in a few days!

I hope we'll resolve it before next release.
thanks again