Failing test: Detection Engine Serverless / Bundled Prebuilte Rules Package API Integration Tests.x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/prebuilt_rules/bundled_prebuilt_rules_package/install_latest_bundled_prebuilt_rules·ts - Detection Engine API - Bundled Prebuilt Rules Package @ess @serverless @skipInQA install_bundled_prebuilt_rules should install prebuilt rules from the package that comes bundled with Kibana

[kibanamachine]

A test failed on a tracked branch

JestAssertionError: expect(received).toBeGreaterThan(expected)

Expected: > 0
Received:   0
    at Context.<anonymous> (install_latest_bundled_prebuilt_rules.ts:75:82)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:73:16) {
  matcherResult: {
    message: '\x1B[2mexpect(\x1B[22m\x1B[31mreceived\x1B[39m\x1B[2m).\x1B[22mtoBeGreaterThan\x1B[2m(\x1B[22m\x1B[32mexpected\x1B[39m\x1B[2m)\x1B[22m\n' +
      '\n' +
      'Expected: > \x1B[32m0\x1B[39m\n' +
      'Received:   \x1B[31m0\x1B[39m',
    pass: false
  }
}

First failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[mistic]

Skipped.

main: 9e087f4

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

This seems to be a legit flake and looks very similar to #172107 (comment). The error happened here on the last line:

      const bundledInstallResponse = await installPrebuiltRulesPackageByVersion(
        es,
        supertest,
        '99.0.0'
      );

      // As opposed to "registry"
      expect(bundledInstallResponse._meta.install_source).toBe('bundled');

      // Refresh ES indices to avoid race conditions between write and reading of indeces
      // See implementation utility function at x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules/install_prebuilt_rules_fleet_package.ts
      await es.indices.refresh({ index: ALL_SAVED_OBJECT_INDICES });

      // Verify that status is updated after package installation
      const statusAfterPackageInstallation = await getPrebuiltRulesStatus(supertest);
      expect(statusAfterPackageInstallation.stats.num_prebuilt_rules_installed).toBe(0);
      expect(statusAfterPackageInstallation.stats.num_prebuilt_rules_to_install).toBeGreaterThan(0);

When both num_prebuilt_rules_installed and num_prebuilt_rules_to_install returned from the endpoint were 0. Which likely means the package wasn't installed by installPrebuiltRulesPackageByVersion or there was a failure with indexing security-rule saved objects or with refreshing that index before the getPrebuiltRulesStatus call.

We need to take a closer look at installPrebuiltRulesPackageByVersion and consider:

• Changing error handling in it. Maybe this endpoint can return 200 even when something fails internally.
• Adding retry logic.
• Making sure await es.indices.refresh works as expected.
• Doing the same with all the other utils that call fleet/epm endpoints, such as installPrebuiltRulesPackageViaFleetAPI.

[banderror]

This test has been already skipped.

[jpdjere]

Fixed and unskipped in: #173998