Merge branch 'main' of github.com:elastic/kibana into reporting/close…

…-chromium-before-compiling-pdf

* 'main' of github.com:elastic/kibana: (121 commits)
  FTR should use the new kibana_system user (elastic#120436)
  [Lens] Temporarily exclude Mosaic/Waffle from the suggestions list (elastic#120488)
  [Reporting] Fix slow CSV with large max size bytes (elastic#120365)
  [CTI] Threat Intel Card on Overview page needs to accommodate Fleet TI integrations -  (elastic#120459)
  [Dashboard] Added KibanaThemeProvider.  (elastic#120122)
  add more rule_registry unit tests (elastic#120323)
  [Lens] update xpack.lens.pie.smallValuesWarningMessage text (elastic#120478)
  [Fleet] Simplified package policy create API, enriching default values (elastic#119739)
  mock `elastic-apm-node` in `@kbn/test` jest preset (elastic#120324)
  [RAC] Rename occurrences of alert_type to rule_type in Infra (elastic#120455)
  [Security Solutions] Removes tech debt of exporting all from linter rule for timeline plugin (elastic#120437)
  [Workplace Search] Add API Keys view to replace Access tokens (elastic#120147)
  [Security Solutions] Removes tech debt of exporting all from linter rule for cases plugin in the server section (elastic#120411)
  Add support for threat.feed.name (elastic#120250)
  [Rule Registry] Rewrite APM registry rules for Observability (elastic#117740)
  [docs] Fix artifact layout formatting (elastic#119386)
  [Workplace Search] Add a technical preview of connecting GitHub via GitHub apps (elastic#119764)
  add osquery notes for 7.16 (elastic#120407)
  chore(NA): splits types from code on @kbn/docs-utils (elastic#120533)
  [Reporting] Decouple screenshotting plugin from the reporting (elastic#120110)
  ...

# Conflicts:
#	x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.test.ts
#	x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.ts
#	x-pack/plugins/reporting/server/lib/screenshots/observable.test.ts
#	x-pack/plugins/reporting/server/lib/screenshots/observable.ts
#	x-pack/plugins/reporting/server/test_helpers/create_mock_browserdriverfactory.ts

.buildkite/pipelines/pull_request/osquery_cypress.yml

@@ -0,0 +1,11 @@
+steps:
+  - command: .buildkite/scripts/steps/functional/osquery_cypress.sh
+    label: 'Osquery Cypress Tests'
+    agents:
+      queue: ci-group-6
+    depends_on: build
+    timeout_in_minutes: 120
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1

.buildkite/scripts/pipelines/pull_request/pipeline.js

@@ -86,6 +86,16 @@ const uploadPipeline = (pipelineContent) => {
       pipeline.push(getPipeline('.buildkite/pipelines/pull_request/fleet_cypress.yml'));
     }
 
+    if (
+      (await doAnyChangesMatch([
+        /^x-pack\/plugins\/osquery/,
+        /^x-pack\/test\/osquery_cypress/,
+      ])) ||
+      process.env.GITHUB_PR_LABELS.includes('ci:all-cypress-suites')
+    ) {
+      pipeline.push(getPipeline('.buildkite/pipelines/pull_request/osquery_cypress.yml'));
+    }
+
     if (await doAnyChangesMatch([/^x-pack\/plugins\/uptime/])) {
       pipeline.push(getPipeline('.buildkite/pipelines/pull_request/uptime.yml'));
     }

.buildkite/scripts/steps/functional/osquery_cypress.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+source .buildkite/scripts/common/util.sh
+
+.buildkite/scripts/bootstrap.sh
+.buildkite/scripts/download_build_artifacts.sh
+
+export JOB=kibana-osquery-cypress
+
+echo "--- Osquery Cypress tests"
+
+cd "$XPACK_DIR"
+
+checks-reporter-with-killswitch "Osquery Cypress Tests" \
+ node scripts/functional_tests \
+   --debug --bail \
+   --kibana-install-dir "$KIBANA_BUILD_LOCATION" \
+   --config test/osquery_cypress/cli_config.ts

.github/CODEOWNERS

@@ -22,7 +22,6 @@
 
 # Vis Editors
 /x-pack/plugins/lens/ @elastic/kibana-vis-editors
-/src/plugins/advanced_settings/ @elastic/kibana-vis-editors
 /src/plugins/charts/ @elastic/kibana-vis-editors
 /src/plugins/vis_default_editor/ @elastic/kibana-vis-editors
 /src/plugins/vis_types/metric/ @elastic/kibana-vis-editors
@@ -263,6 +262,7 @@
 /src/plugins/home/server/*.ts @elastic/kibana-core
 /src/plugins/home/server/services/ @elastic/kibana-core
 /src/plugins/kibana_overview/ @elastic/kibana-core
+/src/plugins/advanced_settings/ @elastic/kibana-core
 /x-pack/plugins/global_search_bar/ @elastic/kibana-core
 #CC# /src/core/server/csp/ @elastic/kibana-core
 #CC# /src/plugins/saved_objects/ @elastic/kibana-core

dev_docs/contributing/standards.mdx

@@ -67,6 +67,14 @@ Every public API should have a release tag specified at the top of it’s docume
 
 Every team should be collecting telemetry metrics on it’s public API usage. This will be important for knowing when it’s safe to make breaking changes. The Core team will be looking into ways to make this easier and an automatic part of registration (see [#112291](https://github.com/elastic/kibana/issues/112291)).
 
+### APM
+
+Kibana server and client are instrumented with APM node and APM RUM clients respectively, tracking serveral types of transactions by default, such as `page-load`, `request`, etc.
+You may introduce custom transactions. Please refer to the [APM documentation](https://www.elastic.co/guide/en/apm/get-started/current/index.html) and follow these guidelines when doing so:
+
+- Use dashed syntax for transaction types and names: `my-transaction-type` and `my-transaction-name`
+- [Refrain from adding too many custom labels](https://www.elastic.co/guide/en/apm/get-started/current/metadata.html)
+
 ### Documentation
 
 Every public API should be documented inside the [docs/api](https://github.com/elastic/kibana/tree/main/docs/api) folder in asciidoc (this content will eventually be migrated to mdx to support the new docs system). If a public REST API is undocumented, you should either document it, or make it internal.

dev_docs/key_concepts/performance.mdx

@@ -3,11 +3,13 @@ id: kibDevPerformance
 slug: /kibana-dev-docs/key-concepts/performance
 title: Performance
 summary: Performance tips for Kibana development.
-date: 2021-09-02
+date: 2021-12-03
 tags: ['kibana', 'onboarding', 'dev', 'performance']
 ---
 
-## Keep Kibana fast
+## Client-side considerations
+
+### Lazy load code
 
 _tl;dr_: Load as much code lazily as possible. Everyone loves snappy
 applications with a responsive UI and hates spinners. Users deserve the
@@ -105,3 +107,15 @@ Many OSS tools allow you to analyze the generated stats file:
   Webpack authors
 - [webpack-visualizer](https://chrisbateman.github.io/webpack-visualizer/)
 - [webpack-bundle-analyzer](https://github.com/webpack-contrib/webpack-bundle-analyzer)
+
+## Server-side considerations
+
+### Don't block the event loop
+
+[Node.js is single threaded](https://nodejs.dev/learn/introduction-to-nodejs) which means a single CPU-intensive server-side, synchronous operation will block any other functionality waiting to execute on the Kibana server. The affects background tasks, like alerts, and search sessions, as well as search requests and page loads.
+
+**When writing code that will run on the server, [don't block the event loop](https://nodejs.org/en/docs/guides/dont-block-the-event-loop/)**. Instead consider:
+
+- Writing async code. For example, leverage [setImmediate](https://nodejs.dev/learn/understanding-setimmediate) inside for loops.
+- Executing logic on the client instead. This may not be a good option if you require a lot of data going back and forth between the server and the client, as that can also slow down the user's experience, especially over slower bandwidth internet connections.
+- Worker threads are also an option if the code doesn't rely on stateful Kibana services. If you are interested in using worker threads, please reach out to a tech-lead before doing so. We will likely want to implement a worker threads pool to ensure worker threads cooperate appropriately.

dev_docs/tutorials/saved_objects.mdx

@@ -252,6 +252,8 @@ Having said that, if a document is encountered that is not in the expected shape
 fail an upgrade than to silently ignore a corrupt document which can cause unexpected behaviour at some future point in time. When such a scenario is encountered,
 the error should be verbose and informative so that the corrupt document can be corrected, if possible.
 
+**WARNING:** Do not attempt to change the `migrationVersion`, `id`, or `type` fields within a migration function, this is not supported.
+
 ### Testing Migrations
 
 Bugs in a migration function cause downtime for our users and therefore have a very high impact. Follow the <DocLink id="kibDevTutorialTestingPlugins" section="saved-objects-migrations" text="Saved Object migrations section in the plugin testing guide"/>.

docs/developer/architecture/core/saved-objects-service.asciidoc

@@ -259,6 +259,9 @@ upgrade. In most scenarios, it is better to fail an upgrade than to silently
 ignore a corrupt document which can cause unexpected behaviour at some future
 point in time.
 
+WARNING: Do not attempt to change the `migrationVersion`, `id`, or `type` fields
+within a migration function, this is not supported.
+
 It is critical that you have extensive tests to ensure that migrations behave
 as expected with all possible input documents. Given how simple it is to test
 all the branch conditions in a migration function and the high impact of a bug

docs/developer/plugin-list.asciidoc

@@ -540,6 +540,11 @@ Elastic.
 |Add tagging capability to saved objects
 
 
+|{kib-repo}blob/{branch}/x-pack/plugins/screenshotting/README.md[screenshotting]
+|This plugin provides functionality to take screenshots of the Kibana pages.
+It uses Chromium and Puppeteer underneath to run the browser in headless mode.
+
+
 |{kib-repo}blob/{branch}/x-pack/plugins/searchprofiler/README.md[searchprofiler]
 |The search profiler consumes the Profile API
 by sending a search API with profile: true enabled in the request body. The response contains

docs/development/core/public/kibana-plugin-core-public.doclinksstart.links.md

@@ -72,13 +72,11 @@ readonly links: {
             readonly indexingDocumentsSchema: string;
             readonly logSettings: string;
             readonly metaEngines: string;
-            readonly nativeAuth: string;
             readonly precisionTuning: string;
             readonly relevanceTuning: string;
             readonly resultSettings: string;
             readonly searchUI: string;
             readonly security: string;
-            readonly standardAuth: string;
             readonly synonyms: string;
             readonly webCrawler: string;
             readonly webCrawlerEventLogs: string;
@@ -90,6 +88,7 @@ readonly links: {
             readonly usersAccess: string;
         };
         readonly workplaceSearch: {
+            readonly apiKeys: string;
             readonly box: string;
             readonly confluenceCloud: string;
             readonly confluenceServer: string;
@@ -105,15 +104,13 @@ readonly links: {
             readonly indexingSchedule: string;
             readonly jiraCloud: string;
             readonly jiraServer: string;
-            readonly nativeAuth: string;
             readonly oneDrive: string;
             readonly permissions: string;
             readonly salesforce: string;
             readonly security: string;
             readonly serviceNow: string;
             readonly sharePoint: string;
             readonly slack: string;
-            readonly standardAuth: string;
             readonly synch: string;
             readonly zendesk: string;
         };

docs/development/core/public/kibana-plugin-core-public.scopedhistory._constructor_.md

@@ -9,13 +9,13 @@ Constructs a new instance of the `ScopedHistory` class
 <b>Signature:</b>
 
 ```typescript
-constructor(parentHistory: History, basePath: string);
+constructor(parentHistory: History<HistoryLocationState>, basePath: string);
 ```
 
 ## Parameters
 
 |  Parameter | Type | Description |
 |  --- | --- | --- |
-|  parentHistory | History |  |
+|  parentHistory | History&lt;HistoryLocationState&gt; |  |
 |  basePath | string |  |
 

docs/development/core/public/kibana-plugin-core-public.scopedhistory.block.md

@@ -9,5 +9,5 @@ Add a block prompt requesting user confirmation when navigating away from the cu
 <b>Signature:</b>
 
 ```typescript
-block: (prompt?: string | boolean | History.TransitionPromptHook<HistoryLocationState> | undefined) => UnregisterCallback;
+block: (prompt?: string | boolean | TransitionPromptHook<HistoryLocationState> | undefined) => UnregisterCallback;
 ```

docs/development/core/public/kibana-plugin-core-public.scopedhistory.createsubhistory.md

@@ -9,5 +9,5 @@ Creates a `ScopedHistory` for a subpath of this `ScopedHistory`<!-- -->. Useful
 <b>Signature:</b>
 
 ```typescript
-createSubHistory: <SubHistoryLocationState = unknown>(basePath: string) => ScopedHistory<SubHistoryLocationState>;
+createSubHistory: (basePath: string) => ScopedHistory<HistoryLocationState>;
 ```

docs/development/core/public/kibana-plugin-core-public.scopedhistory.md

@@ -28,9 +28,9 @@ export declare class ScopedHistory<HistoryLocationState = unknown> implements Hi
 |  Property | Modifiers | Type | Description |
 |  --- | --- | --- | --- |
 |  [action](./kibana-plugin-core-public.scopedhistory.action.md) |  | Action | The last action dispatched on the history stack. |
-|  [block](./kibana-plugin-core-public.scopedhistory.block.md) |  | (prompt?: string \| boolean \| History.TransitionPromptHook&lt;HistoryLocationState&gt; \| undefined) =&gt; UnregisterCallback | Add a block prompt requesting user confirmation when navigating away from the current page. |
+|  [block](./kibana-plugin-core-public.scopedhistory.block.md) |  | (prompt?: string \| boolean \| TransitionPromptHook&lt;HistoryLocationState&gt; \| undefined) =&gt; UnregisterCallback | Add a block prompt requesting user confirmation when navigating away from the current page. |
 |  [createHref](./kibana-plugin-core-public.scopedhistory.createhref.md) |  | (location: LocationDescriptorObject&lt;HistoryLocationState&gt;, { prependBasePath }?: { prependBasePath?: boolean \| undefined; }) =&gt; Href | Creates an href (string) to the location. If <code>prependBasePath</code> is true (default), it will prepend the location's path with the scoped history basePath. |
-|  [createSubHistory](./kibana-plugin-core-public.scopedhistory.createsubhistory.md) |  | &lt;SubHistoryLocationState = unknown&gt;(basePath: string) =&gt; ScopedHistory&lt;SubHistoryLocationState&gt; | Creates a <code>ScopedHistory</code> for a subpath of this <code>ScopedHistory</code>. Useful for applications that may have sub-apps that do not need access to the containing application's history. |
+|  [createSubHistory](./kibana-plugin-core-public.scopedhistory.createsubhistory.md) |  | (basePath: string) =&gt; ScopedHistory&lt;HistoryLocationState&gt; | Creates a <code>ScopedHistory</code> for a subpath of this <code>ScopedHistory</code>. Useful for applications that may have sub-apps that do not need access to the containing application's history. |
 |  [go](./kibana-plugin-core-public.scopedhistory.go.md) |  | (n: number) =&gt; void | Send the user forward or backwards in the history stack. |
 |  [goBack](./kibana-plugin-core-public.scopedhistory.goback.md) |  | () =&gt; void | Send the user one location back in the history stack. Equivalent to calling [ScopedHistory.go(-1)](./kibana-plugin-core-public.scopedhistory.go.md)<!-- -->. If no more entries are available backwards, this is a no-op. |
 |  [goForward](./kibana-plugin-core-public.scopedhistory.goforward.md) |  | () =&gt; void | Send the user one location forward in the history stack. Equivalent to calling [ScopedHistory.go(1)](./kibana-plugin-core-public.scopedhistory.go.md)<!-- -->. If no more entries are available forwards, this is a no-op. |

docs/development/core/server/kibana-plugin-core-server.kibanaresponsefactory.md

@@ -12,14 +12,32 @@ Set of helpers used to create `KibanaResponse` to form HTTP response on an incom
 kibanaResponseFactory: {
     custom: <T extends string | Record<string, any> | Error | Buffer | Stream | {
         message: string | Error;
-        attributes?: Record<string, any> | undefined;
+        attributes?: ResponseErrorAttributes | undefined;
     } | undefined>(options: CustomHttpResponseOptions<T>) => KibanaResponse<T>;
-    badRequest: (options?: ErrorHttpResponseOptions) => KibanaResponse<ResponseError>;
-    unauthorized: (options?: ErrorHttpResponseOptions) => KibanaResponse<ResponseError>;
-    forbidden: (options?: ErrorHttpResponseOptions) => KibanaResponse<ResponseError>;
-    notFound: (options?: ErrorHttpResponseOptions) => KibanaResponse<ResponseError>;
-    conflict: (options?: ErrorHttpResponseOptions) => KibanaResponse<ResponseError>;
-    customError: (options: CustomHttpResponseOptions<ResponseError>) => KibanaResponse<ResponseError>;
+    badRequest: (options?: ErrorHttpResponseOptions) => KibanaResponse<string | Error | {
+        message: string | Error;
+        attributes?: ResponseErrorAttributes | undefined;
+    }>;
+    unauthorized: (options?: ErrorHttpResponseOptions) => KibanaResponse<string | Error | {
+        message: string | Error;
+        attributes?: ResponseErrorAttributes | undefined;
+    }>;
+    forbidden: (options?: ErrorHttpResponseOptions) => KibanaResponse<string | Error | {
+        message: string | Error;
+        attributes?: ResponseErrorAttributes | undefined;
+    }>;
+    notFound: (options?: ErrorHttpResponseOptions) => KibanaResponse<string | Error | {
+        message: string | Error;
+        attributes?: ResponseErrorAttributes | undefined;
+    }>;
+    conflict: (options?: ErrorHttpResponseOptions) => KibanaResponse<string | Error | {
+        message: string | Error;
+        attributes?: ResponseErrorAttributes | undefined;
+    }>;
+    customError: (options: CustomHttpResponseOptions<ResponseError>) => KibanaResponse<string | Error | {
+        message: string | Error;
+        attributes?: ResponseErrorAttributes | undefined;
+    }>;
     redirected: (options: RedirectResponseOptions) => KibanaResponse<string | Record<string, any> | Buffer | Stream>;
     ok: (options?: HttpResponseOptions) => KibanaResponse<string | Record<string, any> | Buffer | Stream>;
     accepted: (options?: HttpResponseOptions) => KibanaResponse<string | Record<string, any> | Buffer | Stream>;

docs/management/connectors/action-types/servicenow-itom.asciidoc

@@ -1,25 +1,31 @@
 [role="xpack"]
 [[servicenow-itom-action-type]]
-=== ServiceNow connector and action
+=== ServiceNow ITOM connector and action
 ++++
 <titleabbrev>ServiceNow ITOM</titleabbrev>
 ++++
 
-The ServiceNow ITOM connector uses the https://docs.servicenow.com/bundle/rome-it-operations-management/page/product/event-management/task/send-events-via-web-service.html[Event API] to create ServiceNow events.
+The {sn} ITOM connector uses the https://docs.servicenow.com/bundle/rome-it-operations-management/page/product/event-management/task/send-events-via-web-service.html[Event API] to create {sn} events.
+
+[float]
+[[servicenow-itom-connector-prerequisites]]
+==== Prerequisites
+Create an integration user in {sn} and assign it the following roles.
+
+* `personalize_choices`: Allows the user to retrieve Choice element options, such as Severity.
+* `evt_mgmt_integration`: Enables integration with external event sources by allowing the user to create events.
 
 [float]
 [[servicenow-itom-connector-configuration]]
 ==== Connector configuration
 
-ServiceNow ITOM connectors have the following configuration properties.
+{sn} ITOM connectors have the following configuration properties.
 
-Name::      The name of the connector. The name is used to identify a  connector in the **Stack Management** UI connector listing, and in the connector list when configuring an action.
-URL::       ServiceNow instance URL.
+Name::      The name of the connector. The name is used to identify a  connector in the **Stack Management** connector listing, and in the connector list when configuring an action.
+URL::       {sn} instance URL.
 Username::  Username for HTTP Basic authentication.
 Password::  Password for HTTP Basic authentication.
 
-The ServiceNow user requires at minimum read, create, and update access to the Event table and read access to the https://docs.servicenow.com/bundle/paris-platform-administration/page/administer/localization/reference/r_ChoicesTable.html[sys_choice]. If you don't provide access to sys_choice, then the choices will not render.
-
 [float]
 [[servicenow-itom-connector-networking-configuration]]
 ==== Connector networking configuration
@@ -55,12 +61,12 @@ Secrets defines sensitive information for the connector type.
 [[define-servicenow-itom-ui]]
 ==== Define connector in Stack Management
 
-Define ServiceNow ITOM connector properties.
+Define {sn} ITOM connector properties.
 
 [role="screenshot"]
 image::management/connectors/images/servicenow-itom-connector.png[ServiceNow ITOM connector]
 
-Test ServiceNow ITOM action parameters.
+Test {sn} ITOM action parameters.
 
 [role="screenshot"]
 image::management/connectors/images/servicenow-itom-params-test.png[ServiceNow ITOM params test]
@@ -69,22 +75,22 @@ image::management/connectors/images/servicenow-itom-params-test.png[ServiceNow I
 [[servicenow-itom-action-configuration]]
 ==== Action configuration
 
-ServiceNow ITOM actions have the following configuration properties.
+{sn} ITOM actions have the following configuration properties.
 
 Source::       The name of the event source type.
 Node::         The Host that the event was triggered for.
 Type::         The type of event.
 Resource::     The name of the resource.
 Metric name::  Name of the metric.
 Source instance (event_class):: Specific instance of the source.
-Message key::  All actions sharing this key will be associated with the same ServiceNow alert. Default value: `<rule ID>:<alert instance ID>`.
+Message key::  All actions sharing this key will be associated with the same {sn} alert. Default value: `<rule ID>:<alert instance ID>`.
 Severity::     The severity of the event.
 Description::  The details about the event.
 
 Refer to https://docs.servicenow.com/bundle/rome-it-operations-management/page/product/event-management/task/send-events-via-web-service.html[ServiceNow documentation] for more information about the properties.
 
 [float]
 [[configuring-servicenow-itom]]
-==== Configure ServiceNow ITOM
+==== Configure {sn} ITOM
 
-ServiceNow offers free https://developer.servicenow.com/dev.do#!/guides/madrid/now-platform/pdi-guide/obtaining-a-pdi[Personal Developer Instances], which you can use to test incidents.
+{sn} offers free https://developer.servicenow.com/dev.do#!/guides/madrid/now-platform/pdi-guide/obtaining-a-pdi[Personal Developer Instances], which you can use to test incidents.