Allow to run multiple independent agents in kubernetes

[mrodm]

Relates #787
Follows #1724

Allow to run several independent agents in kubernetes. For that, agent name daemonset (and related resources) have been named adding the Data Stream and agent Run ID as suffixes. All these Elastic Agent are run in the same kube-system namespace.

It could be added each elastic agent in a different namespace. But this would require to update current packages. It would require to update the kube-state-metrics service URL in the test system configuration since they are running in different namespaces. For instance in the test package from this repository, these changes would be required:

diff --git test/packages/with-kind/kubernetes/data_stream/state_cronjob/_dev/test/system/test-default-config.yml test/packages/with-kind/kubernetes/data_stream/state_cronjob/_dev/test/system/test-default-config.yml
index 0f4bd620..ec4b9f3a 100644
--- test/packages/with-kind/kubernetes/data_stream/state_cronjob/_dev/test/system/test-default-config.yml
+++ test/packages/with-kind/kubernetes/data_stream/state_cronjob/_dev/test/system/test-default-config.yml
@@ -3,4 +3,4 @@ data_stream:
   vars:
     hosts:
       # this is the DNS name of the k8s service for kube-state-metrics deployment
-      - http://kube-state-metrics:8080
+      - http://kube-state-metrics.kube-system:8080
diff --git test/packages/with-kind/kubernetes/data_stream/state_pod/_dev/test/system/test-default-config.yml test/packages/with-kind/kubernetes/data_stream/state_pod/_dev/test/system/test-default-config.yml
index 0f4bd620..ec4b9f3a 100644
--- test/packages/with-kind/kubernetes/data_stream/state_pod/_dev/test/system/test-default-config.yml
+++ test/packages/with-kind/kubernetes/data_stream/state_pod/_dev/test/system/test-default-config.yml
@@ -3,4 +3,4 @@ data_stream:
   vars:
     hosts:
       # this is the DNS name of the k8s service for kube-state-metrics deployment
-      - http://kube-state-metrics:8080
+      - http://kube-state-metrics.kube-system:8080

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: d220ac0

cc @mrodm

[jsoriano]

LGTM!

It could be added each elastic agent in a different namespace. But this would require to update current packages. It would require to update the kube-state-metrics service URL in the test system configuration since they are running in different namespaces.

I think we could do this in any case, so as you prefer.

[mrodm]

It could be added each elastic agent in a different namespace. But this would require to update current packages. It would require to update the kube-state-metrics service URL in the test system configuration since they are running in different namespaces.

I think we could do this in any case, so as you prefer.

Thanks!
I think I will leave it as it is, since kubernetes package has those URLs (kube-state-metrics:8080) in the system test configuration files. If it is changed the namespace where Elastic Agents are deployed, those configuration files should be updated too.