[E2E] TestFleetKubernetesNonRootIntegrationRecipe is failing

[barkbay]

TestFleetKubernetesNonRootIntegrationRecipe seems to have been failing for several days with:

=== RUN   TestFleetKubernetesNonRootIntegrationRecipe/Agent_status_should_be_updated#01
Retries (15m0s timeout): ...........................................................................................................................................................................................................................................................................................................
    step.go:51: 
        	Error Trace:	/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/utils.go:94
        	Error:      	Received unexpected error:
        	            	expected status {Version:8.16.0 ExpectedNodes:0 AvailableNodes:0 Health:green ElasticsearchAssociationsStatus: KibanaAssociationStatus: FleetServerAssociationStatus: ObservedGeneration:0} but got {Version:8.16.0 ExpectedNodes:0 AvailableNodes:0 Health:yellow ElasticsearchAssociationsStatus: KibanaAssociationStatus: FleetServerAssociationStatus: ObservedGeneration:0}
        	Test:       	TestFleetKubernetesNonRootIntegrationRecipe/Agent_status_should_be_updated#01
{"log.level":"error","@timestamp":"2024-12-15T23:39:54.964Z","message":"continuing with additional tests","service.version":"0.0.0-SNAPSHOT+00000000","service.type":"eck","ecs.version":"1.4.0","error":"test Agent status should be updated failed","error.stack_trace":"github.com/elastic/cloud-on-k8s/v2/test/e2e/test.StepList.RunSequential\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/step.go:52\ngithub.com/elastic/cloud-on-k8s/v2/test/e2e/test/helper.RunFile\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/helper/yaml.go:171\ngithub.com/elastic/cloud-on-k8s/v2/test/e2e/agent.runAgentRecipe\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/agent/recipes_test.go:251\ngithub.com/elastic/cloud-on-k8s/v2/test/e2e/agent.TestFleetKubernetesNonRootIntegrationRecipe\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/agent/recipes_test.go:171\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:1689"}
{"log.level":"info","@timestamp":"2024-12-15T23:39:54.968Z","log.logger":"e2e","message":"Running eck-diagnostics","service.version":"0.0.0-SNAPSHOT+00000000","service.type":"eck","ecs.version":"1.4.0","cluster":"eck-e2e-gke-eiie-760","test":"TestFleetKubernetesNonRootIntegrationRecipe","step":"Agent status should be updated"}
    --- FAIL: TestFleetKubernetesNonRootIntegrationRecipe/Agent_status_should_be_updated#01 (900.00s)

[barkbay]

I think the status is not updated as expected because the Agent containers are "oomkilled":

                "containerStatuses": [
                    {
                        "containerID": "containerd://e9a54513e7f1908e6a8da0a94a02bcd82e8fcbd8fc09befa02c15a2ce7e58f94",
                        "image": "docker.elastic.co/beats/elastic-agent:8.16.0",
                        "imageID": "docker.elastic.co/beats/elastic-agent@sha256:0d7ea6e67d7cc30bb5d26f11408df48ab6494b2d6636c92f8e8f4c2113e78f94",
                        "lastState": {
                            "terminated": {
                                "containerID": "containerd://7c8c8693e2c1bdbdc63107ddc0c6f3daaacb2d5d30f850bec11614b17a5c3fa4",
                                "exitCode": 137,
                                "finishedAt": "2024-12-15T23:33:28Z",
                                "reason": "OOMKilled",
                                "startedAt": "2024-12-15T23:32:46Z"
                            }
                        },
                        "name": "agent",
                        "ready": false,
                        "restartCount": 7,
                        "started": false,
                        "state": {
                            "terminated": {
                                "containerID": "containerd://e9a54513e7f1908e6a8da0a94a02bcd82e8fcbd8fc09befa02c15a2ce7e58f94",
                                "exitCode": 137,
                                "finishedAt": "2024-12-15T23:39:56Z",
                                "reason": "OOMKilled",
                                "startedAt": "2024-12-15T23:38:28Z"
                            }
                        }
                    }
                ],

[barkbay]

We already increased the memory a few month ago #8021.

[thbkrkr]

What if we added a check to better highlight this error? Maybe even generalize it to all other Elastic components.

func (b Builder) CheckK8sTestSteps(k *test.K8sClient) test.StepList {
	return test.StepList{
		{
			Name: "Agent pods should not be killed by OOM",
			Test: func(t *testing.T) {
				pods, err := k.GetPods(test.AgentPodListOptions(b.Agent.Namespace, b.Agent.Name)...)
				require.NoError(t, err)
				for _, pod := range pods {
					for _, cs := range pod.Status.ContainerStatuses {
						if cs.State.Terminated.Reason == "OOMKilled" {
							assert.Fail(t, "Agent pod OOMKilled", "pod.name", pod.Name)
						}
					}
				}
			},
		},