[Security Solution] Adds support for testing of prerelease detection …

…rules (#148426)

## Summary

Resolves #147466
Resolves #112910

* Updates `useUpgradeSecurityPackages` hook to install the `prerelease` version of the `endpoint` and `security_detection_engine` packages if the current branch is `main` or build is `-SNAPSHOT` (to ensure PR's are testing against the latest to-be-released packages)
* Adds new `kibana.yml` configuration `xpack.securitySolution.prebuiltRulesPackageVersion` for specifying the version of the `security_detection_engine` package to install within the client-side logic of the `useUpgradeSecurityPackages` hook
* Adds FTR helpers for consuming the `xpack.securitySolution.prebuiltRulesPackageVersion` configuration from the `kbnServerArgs` and for installing a specific detection rules package version [c467762](c467762).
* Regenerated docs
* Unskips `useUpgradeSecurityPackages` tests from [#112910](#112910)

Note: I added jest tests for the `useUpgradeSecurityPackages` changes, however didn't find a reasonable way to test the `prebuiltRulesPackageVersion` configuration addition via FTR's, so ended up testing that manually by running a local `package-registry` and serving up two different versions of the `security_detection_engine` package (`8.3.1`/`8.4.1`) and specifying 

> xpack.securitySolution.prebuiltRulesPackageVersion: '8.3.1'

in my `kibana.dev.yml` to try and install the previous version. This initially failed as fleet would say the package is `out-of-date`

<p align="center">
  <img width="700" src="https://user-images.githubusercontent.com/2946766/211948816-69860629-6db0-4007-8786-3b08f7312baf.png" />
</p>

Since there was a higher version with the same `kibana.version` requirement: `kibana.version: ^8.4.0`. Modifying this for the higher version to `^8.9.0` then allowed for the installation of the `8.3.1` as specified in the `prebuiltRulesPackageVersion` setting:

<p align="center">
  <img width="700" src="https://user-images.githubusercontent.com/2946766/211946889-030c2fdd-6c7d-4124-a1dc-003b54982311.png" />
</p>

<p align="center">
  <img width="700" src="https://user-images.githubusercontent.com/2946766/211948135-03163b0f-b1c2-435a-b91f-c3cbbe028053.png" />
</p>

As [mentioned](#148426 (comment)) by @xcrzx, I ended up adding `force:true` to the individual install request to get around this limitation and to have a better testing experience within Cypress.

Note II: When using the `prebuiltRulesPackageVersion` setting, since this is used for updates initiated from the client and not on kibana start like the `fleet_package.json` (added in #143839), you will have to uninstall the package that was installed on start-up for this to be successful. 

Note III: When wanting to run the Cypress tests against a specific package version, be sure to update the cypress FTR configuration [cf3a83f](cf3a83f).

### Checklist

Delete any items that are not applicable to this PR.

- [X] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

api_docs/security_solution.devdocs.json

@@ -45,13 +45,44 @@
         "deprecated": false,
         "trackAdoption": false,
         "children": [
+          {
+            "parentPluginId": "securitySolution",
+            "id": "def-public.Plugin.kibanaBranch",
+            "type": "string",
+            "tags": [],
+            "label": "kibanaBranch",
+            "description": [
+              "\nThe current Kibana branch. e.g. 'main'"
+            ],
+            "path": "x-pack/plugins/security_solution/public/plugin.tsx",
+            "deprecated": false,
+            "trackAdoption": false
+          },
           {
             "parentPluginId": "securitySolution",
             "id": "def-public.Plugin.kibanaVersion",
             "type": "string",
             "tags": [],
             "label": "kibanaVersion",
-            "description": [],
+            "description": [
+              "\nThe current Kibana version. e.g. '8.0.0' or '8.0.0-SNAPSHOT'"
+            ],
+            "path": "x-pack/plugins/security_solution/public/plugin.tsx",
+            "deprecated": false,
+            "trackAdoption": false
+          },
+          {
+            "parentPluginId": "securitySolution",
+            "id": "def-public.Plugin.prebuiltRulesPackageVersion",
+            "type": "string",
+            "tags": [],
+            "label": "prebuiltRulesPackageVersion",
+            "description": [
+              "\nFor internal use. Specify which version of the Detection Rules fleet package to install\nwhen upgrading rules. If not provided, the latest compatible package will be installed,\nor if running from a dev environment or -SNAPSHOT build, the latest pre-release package\nwill be used (if fleet is available or not within an airgapped environment).\n\nNote: This is for `upgrade only`, which occurs by means of the `useUpgradeSecurityPackages`\nhook when navigating to a Security Solution page. The package version specified in\n`fleet_packages.json` in project root will always be installed first on Kibana start if\nthe package is not already installed."
+            ],
+            "signature": [
+              "string | undefined"
+            ],
             "path": "x-pack/plugins/security_solution/public/plugin.tsx",
             "deprecated": false,
             "trackAdoption": false
@@ -2007,7 +2038,7 @@
         "label": "ConfigType",
         "description": [],
         "signature": [
-          "Readonly<{} & { signalsIndex: string; maxRuleImportExportSize: number; maxRuleImportPayloadBytes: number; maxTimelineImportExportSize: number; maxTimelineImportPayloadBytes: number; alertMergeStrategy: \"allFields\" | \"missingFields\" | \"noFields\"; alertIgnoreFields: string[]; enableExperimental: string[]; packagerTaskInterval: string; }> & { experimentalFeatures: ",
+          "Readonly<{ prebuiltRulesPackageVersion?: string | undefined; } & { signalsIndex: string; maxRuleImportExportSize: number; maxRuleImportPayloadBytes: number; maxTimelineImportExportSize: number; maxTimelineImportPayloadBytes: number; alertMergeStrategy: \"allFields\" | \"missingFields\" | \"noFields\"; alertIgnoreFields: string[]; enableExperimental: string[]; packagerTaskInterval: string; }> & { experimentalFeatures: ",
           "ExperimentalFeatures",
           "; }"
         ],

api_docs/security_solution.mdx

@@ -21,7 +21,7 @@ Contact [Security solution](https://github.com/orgs/elastic/teams/security-solut
 
 | Public API count  | Any count | Items lacking comments | Missing exports |
 |-------------------|-----------|------------------------|-----------------|
-| 113 | 0 | 76 | 29 |
+| 115 | 0 | 75 | 29 |
 
 ## Client
 

src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker

@@ -394,6 +394,7 @@ kibana_vars=(
     xpack.securitySolution.maxTimelineImportExportSize
     xpack.securitySolution.maxTimelineImportPayloadBytes
     xpack.securitySolution.packagerTaskInterval
+    xpack.securitySolution.prebuiltRulesPackageVersion
     xpack.spaces.maxSpaces
     xpack.task_manager.max_attempts
     xpack.task_manager.max_poll_inactivity_cycles

test/plugin_functional/test_suites/core_plugins/rendering.ts

@@ -218,6 +218,7 @@ export default function ({ getService }: PluginFunctionalProviderContext) {
         'xpack.security.sameSiteCookies (alternatives)',
         'xpack.security.showInsecureClusterWarning (boolean)',
         'xpack.securitySolution.enableExperimental (array)',
+        'xpack.securitySolution.prebuiltRulesPackageVersion (string)',
         'xpack.snapshot_restore.slm_ui.enabled (boolean)',
         'xpack.snapshot_restore.ui.enabled (boolean)',
         'xpack.trigger_actions_ui.enableExperimental (array)',

x-pack/plugins/security_solution/public/common/hooks/use_upgrade_secuirty_packages.test.tsx

@@ -6,7 +6,7 @@
  */
 
 import React, { memo } from 'react';
-import { useKibana } from '../lib/kibana';
+import { KibanaServices, useKibana } from '../lib/kibana';
 import type { RenderHookResult } from '@testing-library/react-hooks';
 import { renderHook as _renderHook } from '@testing-library/react-hooks';
 import { useUpgradeSecurityPackages } from './use_upgrade_security_packages';
@@ -23,8 +23,11 @@ jest.mock('../components/user_privileges', () => {
 });
 jest.mock('../lib/kibana');
 
-// FLAKY: https://github.com/elastic/kibana/issues/112910
-describe.skip('When using the `useUpgradeSecurityPackages()` hook', () => {
+describe('When using the `useUpgradeSecurityPackages()` hook', () => {
+  const mockGetPrebuiltRulesPackageVersion =
+    KibanaServices.getPrebuiltRulesPackageVersion as jest.Mock;
+  const mockGetKibanaVersion = KibanaServices.getKibanaVersion as jest.Mock;
+  const mockGetKibanaBranch = KibanaServices.getKibanaBranch as jest.Mock;
   let renderResult: RenderHookResult<object, void>;
   let renderHook: () => RenderHookResult<object, void>;
   let kibana: ReturnType<typeof useKibana>;
@@ -43,6 +46,7 @@ describe.skip('When using the `useUpgradeSecurityPackages()` hook', () => {
   });
 
   afterEach(() => {
+    jest.clearAllMocks();
     if (renderResult) {
       renderResult.unmount();
     }
@@ -65,4 +69,104 @@ describe.skip('When using the `useUpgradeSecurityPackages()` hook', () => {
       })
     );
   });
+
+  it('should send upgrade request with prerelease:false if branch is not `main` and build does not include `-SNAPSHOT`', async () => {
+    mockGetKibanaVersion.mockReturnValue('8.0.0');
+    mockGetKibanaBranch.mockReturnValue('release');
+
+    renderHook();
+
+    await renderResult.waitFor(
+      () => (kibana.services.http.post as jest.Mock).mock.calls.length > 0
+    );
+
+    expect(kibana.services.http.post).toHaveBeenCalledWith(
+      `${epmRouteService.getBulkInstallPath()}`,
+      expect.objectContaining({
+        body: '{"packages":["endpoint","security_detection_engine"]}',
+        query: expect.objectContaining({ prerelease: false }),
+      })
+    );
+  });
+
+  it('should send upgrade request with prerelease:true if branch is `main` AND build includes `-SNAPSHOT`', async () => {
+    mockGetKibanaVersion.mockReturnValue('8.0.0-SNAPSHOT');
+    mockGetKibanaBranch.mockReturnValue('main');
+
+    renderHook();
+
+    await renderResult.waitFor(
+      () => (kibana.services.http.post as jest.Mock).mock.calls.length > 0
+    );
+
+    expect(kibana.services.http.post).toHaveBeenCalledWith(
+      `${epmRouteService.getBulkInstallPath()}`,
+      expect.objectContaining({
+        body: '{"packages":["endpoint","security_detection_engine"]}',
+        query: expect.objectContaining({ prerelease: true }),
+      })
+    );
+  });
+
+  it('should send upgrade request with prerelease:true if branch is `release` and build includes `-SNAPSHOT`', async () => {
+    mockGetKibanaVersion.mockReturnValue('8.0.0-SNAPSHOT');
+    mockGetKibanaBranch.mockReturnValue('release');
+
+    renderHook();
+
+    await renderResult.waitFor(
+      () => (kibana.services.http.post as jest.Mock).mock.calls.length > 0
+    );
+
+    expect(kibana.services.http.post).toHaveBeenCalledWith(
+      `${epmRouteService.getBulkInstallPath()}`,
+      expect.objectContaining({
+        body: '{"packages":["endpoint","security_detection_engine"]}',
+        query: expect.objectContaining({ prerelease: true }),
+      })
+    );
+  });
+
+  it('should send upgrade request with prerelease:true if branch is `main` and build does not include `-SNAPSHOT`', async () => {
+    mockGetKibanaVersion.mockReturnValue('8.0.0');
+    mockGetKibanaBranch.mockReturnValue('main');
+
+    renderHook();
+
+    await renderResult.waitFor(
+      () => (kibana.services.http.post as jest.Mock).mock.calls.length > 0
+    );
+
+    expect(kibana.services.http.post).toHaveBeenCalledWith(
+      `${epmRouteService.getBulkInstallPath()}`,
+      expect.objectContaining({
+        body: '{"packages":["endpoint","security_detection_engine"]}',
+        query: expect.objectContaining({ prerelease: true }),
+      })
+    );
+  });
+
+  it('should send separate upgrade requests if prebuiltRulesPackageVersion is provided', async () => {
+    mockGetPrebuiltRulesPackageVersion.mockReturnValue('8.2.1');
+
+    renderHook();
+
+    await renderResult.waitFor(
+      () => (kibana.services.http.post as jest.Mock).mock.calls.length > 0
+    );
+
+    expect(kibana.services.http.post).toHaveBeenNthCalledWith(
+      1,
+      `${epmRouteService.getInstallPath('security_detection_engine', '8.2.1')}`,
+      expect.objectContaining({ query: { prerelease: true } })
+    );
+    expect(kibana.services.http.post).toHaveBeenNthCalledWith(
+      2,
+      `${epmRouteService.getBulkInstallPath()}`,
+      expect.objectContaining({
+        body: expect.stringContaining('endpoint'),
+        query: expect.objectContaining({ prerelease: true }),
+      })
+    );
+  });
 });

x-pack/plugins/security_solution/public/common/hooks/use_upgrade_security_packages.ts

@@ -9,25 +9,53 @@ import { useEffect } from 'react';
 import type { HttpFetchOptions, HttpStart } from '@kbn/core/public';
 import type { BulkInstallPackagesResponse } from '@kbn/fleet-plugin/common';
 import { epmRouteService } from '@kbn/fleet-plugin/common';
-import { useKibana } from '../lib/kibana';
+import type { InstallPackageResponse } from '@kbn/fleet-plugin/common/types';
+import { KibanaServices, useKibana } from '../lib/kibana';
 import { useUserPrivileges } from '../components/user_privileges';
 
 /**
  * Requests that the endpoint and security_detection_engine package be upgraded to the latest version
  *
  * @param http an http client for sending the request
  * @param options an object containing options for the request
+ * @param prebuiltRulesPackageVersion specific version of the prebuilt rules package to install
  */
 const sendUpgradeSecurityPackages = async (
   http: HttpStart,
-  options: HttpFetchOptions = {}
-): Promise<BulkInstallPackagesResponse> => {
-  return http.post<BulkInstallPackagesResponse>(epmRouteService.getBulkInstallPath(), {
-    ...options,
-    body: JSON.stringify({
-      packages: ['endpoint', 'security_detection_engine'],
-    }),
-  });
+  options: HttpFetchOptions = {},
+  prebuiltRulesPackageVersion?: string
+): Promise<void> => {
+  const packages = ['endpoint', 'security_detection_engine'];
+  const requests: Array<Promise<InstallPackageResponse | BulkInstallPackagesResponse>> = [];
+
+  // If `prebuiltRulesPackageVersion` is provided, try to install that version
+  // Must be done as two separate requests as bulk API doesn't support versions
+  if (prebuiltRulesPackageVersion != null) {
+    packages.splice(packages.indexOf('security_detection_engine'), 1);
+    requests.push(
+      http.post<InstallPackageResponse>(
+        epmRouteService.getInstallPath('security_detection_engine', prebuiltRulesPackageVersion),
+        {
+          ...options,
+          body: JSON.stringify({
+            force: true,
+          }),
+        }
+      )
+    );
+  }
+
+  // Note: if `prerelease:true` option is provided, endpoint package will also be installed as prerelease
+  requests.push(
+    http.post<BulkInstallPackagesResponse>(epmRouteService.getBulkInstallPath(), {
+      ...options,
+      body: JSON.stringify({
+        packages,
+      }),
+    })
+  );
+
+  await Promise.allSettled(requests);
 };
 
 export const useUpgradeSecurityPackages = () => {
@@ -50,8 +78,23 @@ export const useUpgradeSecurityPackages = () => {
           // Make sure fleet is initialized first
           await context.services.fleet?.isInitialized();
 
+          // Always install the latest package if in dev env or snapshot build
+          const isPrerelease =
+            KibanaServices.getKibanaVersion().includes('-SNAPSHOT') ||
+            KibanaServices.getKibanaBranch() === 'main';
+
           // ignore the response for now since we aren't notifying the user
-          await sendUpgradeSecurityPackages(context.services.http, { signal });
+          // Note: response would be Promise.allSettled, so must iterate all responses for errors and throw manually
+          await sendUpgradeSecurityPackages(
+            context.services.http,
+            {
+              query: {
+                prerelease: isPrerelease,
+              },
+              signal,
+            },
+            KibanaServices.getPrebuiltRulesPackageVersion()
+          );
         } catch (error) {
           // Ignore Errors, since this should not hinder the user's ability to use the UI
 

x-pack/plugins/security_solution/public/common/lib/kibana/__mocks__/index.ts

@@ -36,6 +36,8 @@ export const KibanaServices = {
     };
   }),
   getKibanaVersion: jest.fn(() => '8.0.0'),
+  getKibanaBranch: jest.fn(() => 'main'),
+  getPrebuiltRulesPackageVersion: jest.fn(() => undefined),
 };
 export const useKibana = jest.fn().mockReturnValue({
   services: {

x-pack/plugins/security_solution/public/common/lib/kibana/services.ts

@@ -12,27 +12,30 @@ type GlobalServices = Pick<CoreStart, 'application' | 'http' | 'uiSettings' | 'n
   Pick<StartPlugins, 'data' | 'unifiedSearch'>;
 
 export class KibanaServices {
+  private static kibanaBranch?: string;
   private static kibanaVersion?: string;
+  private static prebuiltRulesPackageVersion?: string;
   private static services?: GlobalServices;
 
   public static init({
     http,
     application,
     data,
     unifiedSearch,
+    kibanaBranch,
     kibanaVersion,
+    prebuiltRulesPackageVersion,
     uiSettings,
     notifications,
-  }: GlobalServices & { kibanaVersion: string }) {
-    this.services = {
-      application,
-      data,
-      http,
-      uiSettings,
-      unifiedSearch,
-      notifications,
-    };
+  }: GlobalServices & {
+    kibanaBranch: string;
+    kibanaVersion: string;
+    prebuiltRulesPackageVersion?: string;
+  }) {
+    this.services = { application, data, http, uiSettings, unifiedSearch, notifications };
+    this.kibanaBranch = kibanaBranch;
     this.kibanaVersion = kibanaVersion;
+    this.prebuiltRulesPackageVersion = prebuiltRulesPackageVersion;
   }
 
   public static get(): GlobalServices {
@@ -43,6 +46,14 @@ export class KibanaServices {
     return this.services;
   }
 
+  public static getKibanaBranch(): string {
+    if (!this.kibanaBranch) {
+      this.throwUninitializedError();
+    }
+
+    return this.kibanaBranch;
+  }
+
   public static getKibanaVersion(): string {
     if (!this.kibanaVersion) {
       this.throwUninitializedError();
@@ -51,6 +62,10 @@ export class KibanaServices {
     return this.kibanaVersion;
   }
 
+  public static getPrebuiltRulesPackageVersion(): string | undefined {
+    return this.prebuiltRulesPackageVersion;
+  }
+
   private static throwUninitializedError(): never {
     throw new Error(
       'Kibana services not initialized - are you trying to import this module from outside of the SIEM app?'

x-pack/plugins/security_solution/public/common/mock/endpoint/app_context_render.tsx

@@ -292,6 +292,7 @@ export const createAppRootMockRenderer = (): AppContextTestRender => {
   const globalKibanaServicesParams = {
     ...startServices,
     kibanaVersion: '8.0.0',
+    kibanaBranch: 'main',
   };
 
   if (jest.isMockFunction(KibanaServices.get)) {

x-pack/plugins/security_solution/public/common/types.ts

@@ -18,6 +18,7 @@ export interface ServerApiError {
 
 export interface SecuritySolutionUiConfigType {
   enableExperimental: string[];
+  prebuiltRulesPackageVersion?: string;
 }
 
 /**