Add filestream stream IDs to k8s manifests.

[cmacknz]

Each filestream data stream also needs a unique ID, in addition to the input ID. The input ID is used by the agent itself, the stream level ID is what is used in the Filebeat registry.

• Closes 8.7.1: K8s Stand-alone Kubernetes Deployment - Filestream input with ID '' Already Exists #2701

[mergify]

This pull request does not have a backport label. Could you fix it @cmacknz? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-06-07T18:08:10.968+0000

• Duration: 24 min 21 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5983
Skipped	19
Total	6002

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages.

• run integration tests : Run the Elastic Agent Integration tests.

• run end-to-end tests : Generate the packages and run the E2E Tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	98.667% (74/75)	👍
Files	68.605% (177/258)	👍
Classes	67.423% (327/485)	👍
Methods	53.702% (1001/1864)	👍
Lines	39.597% (11462/28947)	👍 0.021
Conditionals	100.0% (0/0)	💚

[ChrsMark]

So we need this ID in streams' level too? How is this different to the one that already exists at

elastic-agent/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml

Line 333 in 409682e

- id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}

?

cc: @gizas @MichaelKatsoulis

[cmacknz]

Explained here: #2573 (comment)

The input level ID is used by the agent to keep track of the input state. For filestream specifically, each data stream becomes an independent instance of the filestream input in Beats. We need the ID at the stream level to give each filestream input in Beats a unique ID.

There is a one to many mapping between a filestream input in the agent policy and the number of filestream inputs created in Filebeat.

[MichaelKatsoulis]

Although, for kubernetes use case it seems like an one to one mapping, as each agent policy filestream input can only contain one stream, as they both point to one pod-container pair.

[cmacknz]

The input itself uses variable substitution here, so the number of inputs would depend on the number of monitored containers wouldn't it?

Either way, the official examples should use a stream level ID as well so that someone modifying it will a correct example to start from.

[gizas]

@cmacknz why we need both pod.name and container.id? Can we keep only container.id as it is unique afterall?

[cmacknz]

I assume because a pod can contain multiple containers, this is the same thing we use for the input level ID.

[ChrsMark]

So reading #2573 (comment) I understand that we need the id also on the streams level. Based on this the change looks good to me but could we add a reference to any docs or the used GH comment to avoid any confusion in the future?

[cmacknz]

Agreed, we don't have a place in the official docs where we document how to configure each input type in the standalone agent. I just pasted a summary of how this works in the reference YAML and the k8s manifests. Hopefully it helps clarify things.

[ChrsMark]

LGTM