[Detection Engine] Update Prebuilt Security Rules Fleet Package Version Reference v8.4.2

[terrancedejesus]

Summary

Updates Fleet packages reference for prebuilt security rules to v8.4.2 from https://github.com/elastic/detection-rules/releases/tag/integration-v8.4.2

Per prebuilt security rules release process, for the latest package a prerelease package will be published to EPR. Following this, a PR is opened to Kibana to point the fleet packages reference to the prerelease for CI testing. Once successful, a production publication of the package is pushed to EPR and the existing Kibana PR is updated with the production package for CI tests to run again.

Checklist

• Prerelease v8.4.2-beta.1 package published to EPR
• Prerelease v8.4.2-beta.1 package CI-tests ran successfully
• Production v8.4.2 package published to EPR
• Commit with updated Fleet packages reference to v8.4.2 production package
• Production v8.4.2 package CI-tests ran successfully

[terrancedejesus]

Buildkite Job is failing for Security Solution test #3. It is unclear to me at the moment what the issue could be but it appears to originate from E2E cypress test for timelines regarding L109 of creation.cy.ts? @xcrzx do you have any thoughts on this test? I am unfamiliar with cypress testing but from what I have read in documentation so far it simulates web browser activity and captures these tests raising errors where found?

Error

Timelines Creates a timeline by clicking untitled timeline from bottom bar can be added notes

Failures in tracked branches: 0

Buildkite Job
https://buildkite.com/elastic/kibana-pull-request/builds/101232#0185c6bf-5eb0-4d2a-aa7a-e59801067c88

AssertionError: Timed out retrying after 150000ms: expected '<span.euiBadge.CountBadge-sc-14igzi1-1.inyvbd.css-6ivbe3-euiBadge>' to have text '2', but the text was '1'
    at Context.eval (webpack:///./tasks/timeline.ts:170:48)

[xcrzx]

Hey @terrancedejesus, thanks for the ping, I'll take a look at the failing tests. Preliminary, the failure looks more like a flakiness. I'll double-check that.

Also, there were a lot of changes previously with regard to the package distribution and testing on the Kibana side, so that all could be pretty confusing 😅 I think we should do a better job of communicating all the changes and how they should work in conjunction. Hence, I wanted to clarify some things bout the release process and how it should look from the Kibana standpoint.

fleet_packages.json

fleet_packages.json - this config tells what version of the detection rules package will be pre-bundled with Kibana in the following stack release. It is needed only for air-gapped environments where the Elastic package registry is not accessible (no OOB updates and other means of package delivery). That means we need to bump the version in fleet_packages.json a couple of days or weeks prior to stack release.

For example, the next stack version, 8.7.0, will be released on Tue, Mar 28. Before that date and before the last build candidate, a PR that bumps the detection rules package in fleet_packages.json to 8.5.1 (according to the current versioning scheme) should be merged. The 8.5.1 package should be already tested with Kibana and be available in the package registry. The Kibana PR should target the main branch; backports are unnecessary.

Prebuilt rules testing with Kibana

@spong did a great job describing the testing process with Kibana in this PR. The new documentation hasn't been released yet, but once it is, it will be available here (it's a matter of days, I believe). He also created an example PR with the changes needed to test a prerelease package with Kibana.

Release process checklist

So to summarize the above, the release process could look like the following:

For OOB updates:

1. Publish a prerelease package to EPR
2. Create a Kibana PR with the changes to the Cypress config to kick off CI tests (example)
3. Once the CI is green, close the PR without merging it
4. Promote the package to the release version by removing the -beta suffix and publishing it to EPR
5. Remove the prerelease package version from EPR

For release updates

1. All the steps listed above
2. Create a Kibana PR with a fleet_packages.json version bump.
3. Merge it before the last BC to ensure that the new version of the package is bundled with Kibana on release.

@terrancedejesus, What would be the best way to incorporate those steps into your release process to ensure that our teams are on the same page regarding the rules testing/Kibana PRs? We could contribute to your documentation, or you could describe the steps yourself, and we will review them after that. What would be a preferable option for you?

[Mikaayenson]

@xcrzx @spong Can the Cypress tests be exposed via a manually triggered GitHub action workflow? That way we don't have to open and close a non merged PR.

[banderror]

@Mikaayenson Garrett is trying to figure out what options are available and if we'd have to implement support for that. Since this might take some time and effort, I'd suggest we stick to opening PRs for the time being.

[xcrzx]

@xcrzx @spong Can the Cypress tests be exposed via a manually triggered GitHub action workflow? That way we don't have to open and close a non merged PR.

I asked the @elastic/security-engineering-productivity team some time ago about that, and that was not possible. But we could open an enhancement request, I guess.

[Mikaayenson]

That makes sense. Thanks for looking into it for the future. ☺️

[terrancedejesus]

@elasticmachine merge upstream

[terrancedejesus]

Hey @terrancedejesus, thanks for the ping, I'll take a look at the failing tests. Preliminary, the failure looks more like a flakiness. I'll double-check that.

@xcrzx thanks for doing so. I really appreciate your willingness to help our team with some of these issues that we are not yet familiar with. It is a great deal of help! I re-ran the test just to see if it would pass again, considering your comment on it being flaky.

Also, there were a lot of changes previously with regard to the package distribution and testing on the Kibana side, so that all could be pretty confusing 😅 I think we should do a better job of communicating all the changes and how they should work in conjunction. Hence, I wanted to clarify some things bout the release process and how it should look from the Kibana standpoint.

Thank you for adding clarification. I will admit the process has been a little confusing so far as we attempt to update our TRaDE documentation for the release process as well as note where enhancements can be made. As a side note, I will need to take more initiative on learning more about Fleet and Kibana's security solution as well as testing so I can communicate better about potential concerns or enhancements.

fleet_packages.json - this config tells what version of the detection rules package will be pre-bundled with Kibana in the following stack release. It is needed only for air-gapped environments where the Elastic package registry is not accessible (no OOB updates and other means of package delivery). That means we need to bump the version in fleet_packages.json a couple of days or weeks prior to stack release. For example, the next stack version, 8.7.0, will be released on Tue, Mar 28. Before that date and before the last build candidate, a PR that bumps the detection rules package in fleet_packages.json to 8.5.1 (according to the current versioning scheme) should be merged. The 8.5.1 package should be already tested with Kibana and be available in the package registry. The Kibana PR should target the main branch; backports are unnecessary.

Thank you for clarifying all of this. The timing is important to note for our documentation to ensure the fleet packages reference aligns with the build candidates. I also noticed per this PR that it is being auto-updated as well and merged into main which doesn't seem to be an issue as long as we reference the production package prior to final BC and release?

@spong did a great job describing the testing process with Kibana in this PR. The new documentation hasn't been released yet, but once it is, it will be available here (it's a matter of days, I believe). He also created an #149081 with the changes needed to test a prerelease package with Kibana.

Right, I remember reading some of this as your team was working on it but may need to revisit it with some more questions for curiosity or learning as I do not understand the full spectrum of how we test the security solution in Kibana or at least what our CI jobs test specifically. Thank you @spong and team for the excellent work!

Release process checklist
So to summarize the above, the release process could look like the following:

For OOB updates:

Publish a prerelease package to EPR
Create a Kibana PR with the changes to the Cypress config to kick off CI tests (#149081)
Once the CI is green, close the PR without merging it
Promote the package to the release version by removing the -beta suffix and publishing it to EPR
Remove the prerelease package version from EPR

Is there concern for prerelease packages to exist in EPR with the right prefix that forces us to remove them? Concerning customers potentially installing prerelease packages and then those packages being removed, we may have rule version conflicts?

Aside from this, our process updates are being documented here until we finalize them and then publish them on our TRaDE wiki. It is a living document so feel free to add comments, questions, thoughts and anything else your team would like. You will notice that the OOB process you stated is pretty similar to what we had in mind after the required changes for package storage v2 and the Kibana FS rule removal. We have an existing TRaDE issue that we are using to track the release process updates and which @spong added some comments already. Aside from our live document, any additional PRs, issues and so fourth we are trying to track there. Glad we are on similar pages!

For release updates

All the steps listed above
Create a Kibana PR with a fleet_packages.json version bump.
Merge it before the last BC to ensure that the new version of the package is bundled with Kibana on release. @terrancedejesus, What would be the best way to incorporate those steps into your release process to ensure that our teams are on the same page regarding the rules testing/Kibana PRs? We could contribute to your documentation, or you could describe the steps yourself, and we will review them after that. What would be a preferable option for you?

This is what TRaDE had in mind so we are glad our steps are similar for now. Prior to the 8.7 release, we wanted to figure out what changes were necessary to our release process. As a result, OOB updates with packages v8.3.4 and v8.4.2 were great starting points that allowed us to run through our process, identify what change is necessary and update our documentation. We would greatly appreciate any additional insight, potential enhancements, etc. in our working document. Once finalized we will publish it to our TRaDE wiki. Our goal is to make releases faster so we can eventually do either weekly or bi-weekly releases and it be a relatively short and effortless process. I think we have the tools to do so, just need some better insight and creativity 😄

[terrancedejesus]

All CI testing has passed for this prerelease package as detailed in #149158. We can release the v8.4.2 production package to EPR and update the fleet packages file accordingly.

[banderror]

@terrancedejesus Shouldn't we explicitly set the 8.4.2-beta.1 version for Cypress tests (example) in this PR to 100% make sure that the CI has run the tests against this specific version? I believe it has because I guess it was the latest one at the time of the run. But I wouldn't rely on that 😅

This is a good point. So on similar PRs, we should update fleet packages reference and adjust the Cypress test xpack value for the security solution to ensure they pass?

[xcrzx]

Shouldn't we explicitly set the 8.4.2-beta.1 version for Cypress tests (example) in this PR to 100% make sure that the CI has run the tests against this specific version? I believe it has because I guess it was the latest one at the time of the run. But I wouldn't rely on that 😅

We should always set the package version explicitly in the x-pack/test/security_solution_cypress/config.ts config to be sure we're testing the correct version. That way, tests will fail if, for example, the package is unavailable in the registry. Without that setting, tests could grab another version and test against it.

[terrancedejesus]

@elasticmachine merge upstream

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 98bed35

Failed CI Steps

• Security Solution Tests #3

Test Failures

• [job] [logs] Security Solution Tests #3 / Timelines Creates a timeline by clicking untitled timeline from bottom bar can be added notes

Metrics [docs]

✅ unchanged

History

• 💛 Build #101536 was flaky ad85423
• 💔 Build #101479 failed d18d9ae
• 💔 Build #101232 failed b0e9cfa

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @terrancedejesus

[xcrzx]

There's a PR created by automation, btw: https://github.com/elastic/kibana/pull/149245/files.

[terrancedejesus]

Closing this issue and tracking the changes in #149245 as a Fleet Code Owner has already approved. Thanks everyone for the additional commentary!