[Fleet] [Security Solution] Install prebuilt rules package using stream-based approach

[xcrzx]

Resolves: #192350

Summary

Implemented stream-based installation of the detection rules package.

Background: The installation of the detection rules package was causing OOM (Out of Memory) errors in Serverless environments where the available memory is limited to 1GB. The root cause of the errors was that during installation, the package was being read and unzipped entirely into memory. Given the large package size, this led to OOMs. To address these memory issues, the following changes were made:

1. Added a branching logic to the installPackageFromRegistry and installPackageByUpload methods, where based on the package name is decided to use streaming or not. Only one security_detection_engine package is currently hardcoded to use streaming.
2. In the state machine then defined a separate set of steps for the stream-based package installation. It is reduced to cover only Kibana assets installation at this stage.
3. A new stepInstallKibanaAssetsWithStreaming step is added to handle assets installation. While this method still reads the package archive into memory (since unzipping from a readable stream is not possible due to the design of the .zip format), the package is unzipped using streams after being read into a buffer. This allows only a small portion of the archive (100 saved objects at a time) to be unpacked into memory, reducing memory usage.
4. The new method also includes several optimizations, such as only removing previously installed assets if they are missing in the new package and using savedObjectClient.bulkCreate instead of the less efficient savedObjectClient.import.

Test environment

1. Prebuilt detection rules package with ~20k saved objects; 118MB zipped.
2. Local package registry.
3. Production build of Kibana running locally with a 700MB max old space limit, pointed to that registry.

Setting up a test environment is not completely straightforward. Here's a rough outline of the steps:

How to test this PR

1. Create a package containing a large number of prebuilt rules.
   1. I used the package-storage repository to find one of the previously released prebuilt rules packages.
   2. Multiplied the number of assets in the package to 20k historical versions.
   3. Built the package using elastic-package build.
2. Start a local package registry serving the built package using elastic-package stack up --services package-registry.
3. Create a production build of Kibana. To speed up the process, unnecessary artifacts can be skipped:

   node scripts/build --skip-cdn-assets --skip-docker-ubi --skip-docker-ubuntu --skip-docker-wolfi --skip-docker-fips
   

4. Provide the built Kibana with a config pointing to the local registry. The config is located in build/default/kibana-9.0.0-SNAPSHOT-darwin-aarch64/config/kibana.yml. You can use the following config:

   csp.strict: false
   xpack.security.encryptionKey: 've4Vohnu oa0Fu9ae Eethee8c oDieg4do Nohrah1u ao9Hu2oh Aeb4Ieyi Aew1aegi'
   xpack.encryptedSavedObjects.encryptionKey: 'Shah7nai Eew6izai Eir7OoW0 Gewi2ief eiSh8woo shoogh7E Quae6hal ce6Oumah'
   
   xpack.fleet.internal.registry.kibanaVersionCheckEnabled: false
   xpack.fleet.registryUrl: https://localhost:8080
   
   elasticsearch:
     username: 'kibana_system'
     password: 'changeme'
     hosts: 'http://localhost:9200'
   

5. Override the Node options Kibana starts with to allow it to connect to the local registry and set the memory limit. For this, you need to edit the build/default/kibana-9.0.0-SNAPSHOT-darwin-aarch64/bin/kibana file:

   NODE_OPTIONS="--no-warnings --max-http-header-size=65536 --unhandled-rejections=warn --dns-result-order=ipv4first --openssl-legacy-provider --max_old_space_size=700 --inspect" NODE_ENV=production NODE_EXTRA_CA_CERTS=~/.elastic-package/profiles/default/certs/ca-cert.pem exec "${NODE}" "${DIR}/src/cli/dist" "${@}"
   

6. Navigate to the build folder: build/default/kibana-9.0.0-SNAPSHOT-darwin-aarch64.
7. Start Kibana using ./bin/kibana.
8. Kibana is now running in debug mode, with the debugger started on port 9229. You can connect to it using VS Code's debug config or Chrome's DevTools.
9. Now you can install prebuilt detection rules by calling the POST /internal/detection_engine/prebuilt_rules/_bootstrap endpoint, which uses the new streaming installation under the hood.

Test results locally

Without the streaming approach

Guaranteed OOM. Even smaller packages, up to 10k rules, caused sporadic OOM errors. So for comparison, tested the package installation without memory limits.

1. Heap memory usage spikes up to 2.5GB
2. External memory consumes up to 450 Mb, which is four times the archive size
3. RSS (Resident Set Size) exceeds 4.5GB

With the streaming approach

No OOM errors observed. The memory consumption chart looks like the following:

1. Heap memory remains stable, around 450MB, without any spikes.
2. External memory jumps to around 250MB at the beginning of the installation, then drops to around 120MB, which is roughly equal to the package archive size. I couldn't determine why the external memory consumption exceeds the package size by 2x when the installation starts. I checked the code for places where the package might be loaded into memory twice but found nothing suspicious. This might be worth investigating further.
3. RSS remains stable, peaking slightly above 1GB. I believe this is the upper limit for a package that can be handled without errors in a Serverless environment, where the memory limit is dictated by pod-level settings rather than Node settings and is set to 1GB. I'll verify this on a real Serverless instance to confirm.

Test results on Serverless

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[nchaulet]

what happen if an error happens during the onEntry call here? should we handle it and do we have to manually close the zipFile?

[xcrzx]

Good catch! Added a try ... finally block to continue iterating on an error. That matches the behavior of untarBuffer

[nchaulet]

Tested locally and it seems to works well LGTM 🚀

It will be great to add an api integration test that install a streaming package, maybe we can add a security_detection_engine with a few rules in x-pack/test/fleet_api_integration/apis/fixtures/test_packages and try to install it? it could also come as a follow up issue/PR

[xcrzx]

It will be great to add an api integration test that install a streaming package, maybe we can add a security_detection_engine with a few rules in x-pack/test/fleet_api_integration/apis/fixtures/test_packages and try to install it?

Added an integration test x-pack/test/fleet_api_integration/apis/epm/install_with_streaming.ts 👍

[elasticmachine]

⏳ Build in-progress

• Buildkite Build
• Commit: d6a3036
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-195888-d6a3036501e7
• Security Deployment

History

• 💔 Build #247945 failed e374721
• 💚 Build #246463 succeeded 75eea3e
• 💔 Build #246412 failed 94a3e50b42f1ac254ffae69f2b5e03cf1295eddc
• 💚 Build #246027 succeeded 1b97750028df15d3ac4377da0ebe077d771bf923
• 💚 Build #245887 succeeded a9e5451b45ac0f59485e99079278049c514d5533

cc @xcrzx

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11683891974

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation