Failing test: X-Pack Cloud Security Posture Functional Tests.x-pack/test/cloud_security_posture_functional/pages/findings·ts - Cloud Security Posture Findings Page "before all" hook in "Findings Page"

[kibanamachine]

A test failed on a tracked branch

Error: timed out waiting for Findings table to be loaded -- last error: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="findings_table"])
Wait timed out after 10029ms
    at /var/lib/buildkite-agent/builds/kb-n2-4-spot-ce614a930618b89c/elastic/kibana-on-merge/kibana/node_modules/selenium-webdriver/lib/webdriver.js:907:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at onFailure (node_modules/@kbn/ftr-common-functional-services/target_node/services/retry/retry_for_truthy.js:34:13)
    at retryForSuccess (node_modules/@kbn/ftr-common-functional-services/target_node/services/retry/retry_for_success.js:55:13)
    at retryForTruthy (node_modules/@kbn/ftr-common-functional-services/target_node/services/retry/retry_for_truthy.js:24:3)
    at RetryService.waitFor (node_modules/@kbn/ftr-common-functional-services/target_node/services/retry/retry.js:52:5)
    at Context.<anonymous> (x-pack/test/cloud_security_posture_functional/pages/findings.ts:46:7)
    at Object.apply (node_modules/@kbn/test/target_node/src/functional_test_runner/lib/mocha/wrap_function.js:78:16)

First failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[spalger]

/skip

[kibanamachine]

Skipped

main: 0fd9c9f

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security Posture)

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[orouz]

initial findings show that the test suite started failing with an error about a missing .fleet-agents index in 549ab35. seems like it started happening after a9166da, as things work like before when using its parent - efb7cdd.

looks like getAgentStatusForAgentPolicy changed to always throw when querying AGENT_INDEX

it used to just log:

kibana/x-pack/plugins/fleet/server/services/agents/status.ts

Lines 69 to 77 in efb7cdd

	} catch (error) {
	if (error.statusCode === 404) {
	appContextService
	.getLogger()
	.debug('Index .fleet-agents does not exist yet, skipping point in time.');
	} else {
	throw error;
	}
	}

now it always throws:

kibana/x-pack/plugins/fleet/server/services/agents/status.ts

Lines 110 to 113 in a9166da

	} catch (error) {
	logger.error(`Error getting agent statuses: ${error}`);
	throw error;
	}

this explains the error we see in both tests (FTR) and local development, as in both the CSP /status api returns an error:

a quick fix in our plugin is to add a try/catch to wrap this:

kibana/x-pack/plugins/cloud_security_posture/server/lib/fleet_util.ts

Line 38 in 1c3dc4a

export const getAgentStatusesByAgentPolicies = async (

but even if the above is correct, i still can't tell why it started failing on 549ab35 and not directly after a9166da. so there's probably more to it.

one thing i noticed which is odd is that after elastic/integrations#4752 (comment) was released to EPR (Dec 4), all of our FTR tests passed, but started logging fleet setup failed and on Dec 22 they started failing, but logged fleet setup completed

[orouz]

ok, the root issue here is that our FTR test ran with latest as the package to install, and that's a moving target.

so timeline is like this:

• Dec 4: [email protected] released to EPR

  • fleet setup initially fails to validate the package due to new missing required vars which are not specified in FTR config (can't be). package is installed, but no package/agent policies are created. FTR tests pass because we have an installation and we indexed findings, which is sufficient to view our pages. we don't query agent policies because we don't have any, since the setup failed.

• Dec 22, 12am: commit merged that will throw when we query for agent policies

  • it passes because we stopped triggering that code branch when fleet setup started failing, without creating policies, so we didn't query for them.

• Dec 22, 1:59 PM: [email protected] released to EPR,

  • makes it latest package, even though it targets ^8.6.0 and [email protected] targets ^8.7.0
  • this package does not include the required vars that fail the fleet setup, so it completes successfully (with package/agent policies)

• Dec 22, 2:22 PM: first CI failure,

  • uses the new package [email protected]
  • fleet setup completes with preconfigured policies
  • we query for agent policies which now throws

which explains why our FTR tests passed in a9166da and started failing in 549ab35.

initial fix is to use a fixed version number hard-coded in our FTR test, which will make them only run with that specific version. this means:

• changes to our integration won't suddenly break FTR tests
• changes to our integration aren't automatically tested. authors need to:
  • change the fixed version in our FTR tests
  • run our FTR tests

this seems like a simple solution, although a bit annoying in practice. there is also an option to always use latest package but run a custom package registry. at the time being, this wasn't explored any further.

[kibanamachine]

New failure: CI Build - main