[Metricbeat Autodiscover] Provider Kubernetes always creates watcher for namespaces and nodes

[constanca-m]

Issue

add_resource_metadata is enabled at all times for two resources, namespaces and nodes.

There is an option to disable them (warning: hints should also be disabled, otherwise this will not work). For simplicity, I will only use namespace for this issue:

add_resource_metadata:
  namespace.enabled: false

However, when creating the watcher, we don't check if it is enabled or not:

beats/libbeat/autodiscover/providers/kubernetes/pod.go

Line 110 in cc39376

namespaceWatcher, err := kubernetes.NewNamedWatcher("namespace", client, &kubernetes.Namespace{}, kubernetes.WatchOptions{

And only later when it is time to add events, do we check for that:

beats/libbeat/autodiscover/providers/kubernetes/pod.go

Lines 160 to 163 in cc39376

	if namespaceWatcher != nil && (config.Hints.Enabled() || metaConf.Namespace.Enabled()) {
	updater := kubernetes.NewNamespacePodUpdater(p.unlockedUpdate, watcher.Store(), &p.crossUpdate)
	namespaceWatcher.AddEventHandler(updater)
	}

The problem with this approach is that we always need to have permissions to list namespaces within the cluster scope, when we shouldn't force the user to have them. But if they don't, we always face the error:

W1122 12:17:27.683394      23 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:metricbeat" cannot list resource "namespaces" in API group "" at the cluster scope

We face this issue when resource is set to pod (default) and service.

We also need to prevent namespace/node watchers from being created when using enrichers.

To complete

• Update code to stop watchers from starting unless add_resource_metadata or hints are enabled:
  • Beats: [Metricbeat][Kubernetes] Remove mandatory permissions for namespace and node #38762
  • Elastic agent: [Kubernetes] Check AddResourceMetadata and hints before starting namespace and node watchers elastic-agent#4618

[jeanfabrice]

Hey @constanca-m 👋
Any updates on this issue? Is #37243 the right issue to track for progression?
At the moment, it looks like 7.10.2 is the latest release that is free from this bug. But 7.10.2 is EOL. Thanks!

[constanca-m]

Hi @jeanfabrice. Yes, #37243 is the right issue to track right now. It takes a long time to merge because it modifies a lot of sensitive code.