Merge branch 'main' into issues/65538

docs/developer/getting-started/debugging.asciidoc

@@ -130,71 +130,3 @@ Once you're finished, you can stop Kibana normally, then stop the {es} and APM s
 ----
 ./scripts/compose.py stop
 ----
-
-=== Using {kib} server logs
-{kib} Logs is a great way to see what's going on in your application and to debug performance issues. Navigating through a large number of generated logs can be overwhelming, and following are some techniques that you can use to optimize the process.
-
-Start by defining a problem area that you are interested in. For example, you might be interested in seeing how a particular {kib} Plugin is performing, so no need to gather logs for all of {kib}. Or you might want to focus on a particular feature, such as requests from the {kib} server to the {es} server.
-Depending on your needs, you can configure {kib} to generate logs for a specific feature.
-[source,yml]
-----
-logging:
-  appenders:
-    file:
-      type: file
-      fileName: ./kibana.log
-      layout:
-        type: json
-
-### gather all the Kibana logs into a file
-logging.root:
-    appenders: [file]
-    level: all
-
-### or gather a subset of the logs
-logging.loggers:
-  ### responses to an HTTP request
-  - name: http.server.response
-    level: debug
-    appenders: [file]
-  ### result of a query to the Elasticsearch server
-  - name: elasticsearch.query
-    level: debug
-    appenders: [file]
-  ### logs generated by my plugin
-  - name: plugins.myPlugin
-    level: debug
-    appenders: [file]
-----
-WARNING: Kibana's `file` appender is configured to produce logs in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format. It's the only format that includes the meta information necessary for https://www.elastic.co/guide/en/apm/agent/nodejs/current/log-correlation.html[log correlation] out-of-the-box.
-
-The next step is to define what https://www.elastic.co/observability[observability tools] are available. 
-For a better experience, set up an https://www.elastic.co/guide/en/apm/get-started/current/observability-integrations.html[Observability integration] provided by Elastic to debug your application with the <<debugging-logs-apm-ui, APM UI.>>
-To debug something quickly without setting up additional tooling, you can work with <<plain-kibana-logs, the plain {kib} logs.>>
-
-[[debugging-logs-apm-ui]]
-==== APM UI
-*Prerequisites* {kib} logs are configured to be in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format to include tracing identifiers.
-
-To debug {kib} with the APM UI, you must set up the APM infrastructure. You can find instructions for the setup process
-https://www.elastic.co/guide/en/apm/get-started/current/observability-integrations.html[on the Observability integrations page].
-
-Once you set up the APM infrastructure, you can enable the APM agent and put {kib} under load to collect APM events. To analyze the collected metrics and logs, use the APM UI as demonstrated https://www.elastic.co/guide/en/kibana/master/transactions.html#transaction-trace-sample[in the docs].
-
-[[plain-kibana-logs]]
-==== Plain {kib} logs
-*Prerequisites* {kib} logs are configured to be in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format to include tracing identifiers.
-
-Open {kib} Logs and search for an operation you are interested in.
-For example, suppose you want to investigate the response times for queries to the `/api/telemetry/v2/clusters/_stats` {kib} endpoint.
-Open Kibana Logs and search for the HTTP server response for the endpoint. It looks similar to the following (some fields are omitted for brevity).
-[source,json]
-----
-{
-  "message":"POST /api/telemetry/v2/clusters/_stats 200 1014ms - 43.2KB",
-  "log":{"level":"DEBUG","logger":"http.server.response"},
-  "trace":{"id":"9b99131a6f66587971ef085ef97dfd07"},
-  "transaction":{"id":"d0c5bbf14f5febca"}
-}
-----
-You are interested in the https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html#field-trace-id[trace.id] field, which is a unique identifier of a trace. The `trace.id` provides a way to group multiple events, like transactions, which belong together. You can search for `"trace":{"id":"9b99131a6f66587971ef085ef97dfd07"}` to get all the logs that belong to the same trace. This enables you to see how many {es} requests were triggered during the `9b99131a6f66587971ef085ef97dfd07` trace, what they looked like, what {es} endpoints were hit, and so on.

docs/setup/upgrade.asciidoc

@@ -44,13 +44,20 @@ a|
 [[upgrade-before-you-begin]]
 === Before you begin
 
-WARNING: {kib} automatically runs upgrade migrations when required. To roll back to an earlier version in case of an upgrade failure, you **must** have a {ref}/snapshot-restore.html[backup snapshot] available. This snapshot must include the `kibana` feature state or all `kibana*` indices. For more information see <<upgrade-migrations, upgrade migrations>>.
+[WARNING]
+====
+{kib} automatically runs upgrade migrations when required. To roll back to an
+earlier version in case of an upgrade failure, you **must** have a
+{ref}/snapshot-restore.html[backup snapshot] that includes the `kibana` feature
+state. Snapshots include this feature state by default.
+
+For more information, refer to <<upgrade-migrations, upgrade migrations>>.
+====
 
 Before you upgrade {kib}:
 
 * Consult the <<breaking-changes,breaking changes>>.
-* {ref}/snapshots-take-snapshot.html[Take a snapshot] of your data. To roll back to an earlier version, the snapshot must include the `kibana` feature state or all `.kibana*` indices. 
-* Although not a requirement for rollbacks, we recommend taking a snapshot of all {kib} indices created by the plugins you use such as the `.reporting*` indices created by the reporting plugin.  
+* {ref}/snapshots-take-snapshot.html[Take a snapshot] of your data. To roll back to an earlier version, the snapshot must include the `kibana` feature state.
 * Before you upgrade production servers, test the upgrades in a dev environment.
 * See <<preventing-migration-failures, preventing migration failures>> for common reasons upgrades fail and how to prevent these.
 * If you are using custom plugins, check that a compatible version is

docs/setup/upgrade/upgrade-migrations.asciidoc

@@ -151,17 +151,18 @@ In order to rollback after a failed upgrade migration, the saved object indices
 [float]
 ===== Rollback by restoring a backup snapshot:
 
-1. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state or all `.kibana*` indices.
+1. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state.
+   Snapshots include this feature state by default.
 2. Shutdown all {kib} instances to be 100% sure that there are no instances currently performing a migration.
 3. Delete all saved object indices with `DELETE /.kibana*`
-4. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state or all `.kibana* indices and their aliases from the snapshot.
+4. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot.
 5. Start up all {kib} instances on the older version you wish to rollback to.
 
 [float]
 ===== (Not recommended) Rollback without a backup snapshot:
 
 1. Shutdown all {kib} instances to be 100% sure that there are no {kib} instances currently performing a migration.
-2. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state or all `.kibana*` indices.
+2. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. Snapshots include this feature state by default.
 3. Delete the version specific indices created by the failed upgrade migration. E.g. if you wish to rollback from a failed upgrade to v7.12.0 `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`
 4. Inspect the output of `GET /_cat/aliases`. If either the `.kibana` and/or `.kibana_task_manager` alias is missing, these will have to be created manually. Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index. E.g. if the `.kibana` alias was missing and the latest index is `.kibana_3` create a new alias with `POST /.kibana_3/_aliases/.kibana`.
 5. Remove the write block from the rollback indices. `PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`

docs/user/index.asciidoc

@@ -45,3 +45,5 @@ include::management.asciidoc[]
 include::api.asciidoc[]
 
 include::plugins.asciidoc[]
+
+include::troubleshooting.asciidoc[]

docs/user/troubleshooting.asciidoc

@@ -0,0 +1,70 @@
+[[kibana-troubleshooting]]
+== Troubleshooting
+
+=== Using {kib} server logs
+{kib} Logs is a great way to see what's going on in your application and to debug performance issues. Navigating through a large number of generated logs can be overwhelming, and following are some techniques that you can use to optimize the process.
+
+Start by defining a problem area that you are interested in. For example, you might be interested in seeing how a particular {kib} Plugin is performing, so no need to gather logs for all of {kib}. Or you might want to focus on a particular feature, such as requests from the {kib} server to the {es} server.
+Depending on your needs, you can configure {kib} to generate logs for a specific feature.
+[source,yml]
+----
+logging:
+  appenders:
+    file:
+      type: file
+      fileName: ./kibana.log
+      layout:
+        type: json
+
+### gather all the Kibana logs into a file
+logging.root:
+    appenders: [file]
+    level: all
+
+### or gather a subset of the logs
+logging.loggers:
+  ### responses to an HTTP request
+  - name: http.server.response
+    level: debug
+    appenders: [file]
+  ### result of a query to the Elasticsearch server
+  - name: elasticsearch.query
+    level: debug
+    appenders: [file]
+  ### logs generated by my plugin
+  - name: plugins.myPlugin
+    level: debug
+    appenders: [file]
+----
+WARNING: Kibana's `file` appender is configured to produce logs in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format. It's the only format that includes the meta information necessary for https://www.elastic.co/guide/en/apm/agent/nodejs/current/log-correlation.html[log correlation] out-of-the-box.
+
+The next step is to define what https://www.elastic.co/observability[observability tools] are available. 
+For a better experience, set up an https://www.elastic.co/guide/en/apm/get-started/current/observability-integrations.html[Observability integration] provided by Elastic to debug your application with the <<debugging-logs-apm-ui, APM UI.>>
+To debug something quickly without setting up additional tooling, you can work with <<plain-kibana-logs, the plain {kib} logs.>>
+
+[[debugging-logs-apm-ui]]
+==== APM UI
+*Prerequisites* {kib} logs are configured to be in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format to include tracing identifiers.
+
+To debug {kib} with the APM UI, you must set up the APM infrastructure. You can find instructions for the setup process
+https://www.elastic.co/guide/en/apm/get-started/current/observability-integrations.html[on the Observability integrations page].
+
+Once you set up the APM infrastructure, you can enable the APM agent and put {kib} under load to collect APM events. To analyze the collected metrics and logs, use the APM UI as demonstrated https://www.elastic.co/guide/en/kibana/master/transactions.html#transaction-trace-sample[in the docs].
+
+[[plain-kibana-logs]]
+==== Plain {kib} logs
+*Prerequisites* {kib} logs are configured to be in https://www.elastic.co/guide/en/ecs/master/ecs-reference.html[ECS JSON] format to include tracing identifiers.
+
+Open {kib} Logs and search for an operation you are interested in.
+For example, suppose you want to investigate the response times for queries to the `/api/telemetry/v2/clusters/_stats` {kib} endpoint.
+Open Kibana Logs and search for the HTTP server response for the endpoint. It looks similar to the following (some fields are omitted for brevity).
+[source,json]
+----
+{
+  "message":"POST /api/telemetry/v2/clusters/_stats 200 1014ms - 43.2KB",
+  "log":{"level":"DEBUG","logger":"http.server.response"},
+  "trace":{"id":"9b99131a6f66587971ef085ef97dfd07"},
+  "transaction":{"id":"d0c5bbf14f5febca"}
+}
+----
+You are interested in the https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html#field-trace-id[trace.id] field, which is a unique identifier of a trace. The `trace.id` provides a way to group multiple events, like transactions, which belong together. You can search for `"trace":{"id":"9b99131a6f66587971ef085ef97dfd07"}` to get all the logs that belong to the same trace. This enables you to see how many {es} requests were triggered during the `9b99131a6f66587971ef085ef97dfd07` trace, what they looked like, what {es} endpoints were hit, and so on.

package.json

@@ -107,7 +107,7 @@
     "@elastic/datemath": "link:bazel-bin/packages/elastic-datemath",
     "@elastic/elasticsearch": "npm:@elastic/elasticsearch-canary@^8.0.0-canary.35",
     "@elastic/ems-client": "8.0.0",
-    "@elastic/eui": "41.0.0",
+    "@elastic/eui": "41.2.3",
     "@elastic/filesaver": "1.1.2",
     "@elastic/node-crypto": "1.2.1",
     "@elastic/numeral": "^2.5.1",
@@ -369,7 +369,7 @@
     "redux-thunks": "^1.0.0",
     "regenerator-runtime": "^0.13.3",
     "remark-parse": "^8.0.3",
-    "remark-stringify": "^9.0.0",
+    "remark-stringify": "^8.0.3",
     "require-in-the-middle": "^5.1.0",
     "reselect": "^4.0.0",
     "resize-observer-polyfill": "^1.5.1",
@@ -521,7 +521,6 @@
     "@types/ejs": "^3.0.6",
     "@types/elastic__apm-synthtrace": "link:bazel-bin/packages/elastic-apm-synthtrace/npm_module_types",
     "@types/elastic__datemath": "link:bazel-bin/packages/elastic-datemath/npm_module_types",
-    "@types/elasticsearch": "^5.0.33",
     "@types/enzyme": "^3.10.8",
     "@types/eslint": "^7.28.0",
     "@types/express": "^4.17.13",
@@ -571,6 +570,7 @@
     "@types/kbn__dev-utils": "link:bazel-bin/packages/kbn-dev-utils/npm_module_types",
     "@types/kbn__docs-utils": "link:bazel-bin/packages/kbn-docs-utils/npm_module_types",
     "@types/kbn__es-archiver": "link:bazel-bin/packages/kbn-es-archiver/npm_module_types",
+    "@types/kbn__es-query": "link:bazel-bin/packages/kbn-es-query/npm_module_types",
     "@types/kbn__i18n": "link:bazel-bin/packages/kbn-i18n/npm_module_types",
     "@types/kbn__i18n-react": "link:bazel-bin/packages/kbn-i18n-react/npm_module_types",
     "@types/license-checker": "15.0.0",

packages/BUILD.bazel

@@ -89,6 +89,7 @@ filegroup(
       "//packages/kbn-dev-utils:build_types",
       "//packages/kbn-docs-utils:build_types",
       "//packages/kbn-es-archiver:build_types",
+      "//packages/kbn-es-query:build_types",
       "//packages/kbn-i18n:build_types",
       "//packages/kbn-i18n-react:build_types",
   ],

packages/kbn-config-schema/src/byte_size_value/index.test.ts

@@ -30,6 +30,11 @@ describe('parsing units', () => {
     expect(ByteSizeValue.parse('1gb').getValueInBytes()).toBe(1073741824);
   });
 
+  test('case insensitive units', () => {
+    expect(ByteSizeValue.parse('1KB').getValueInBytes()).toBe(1024);
+    expect(ByteSizeValue.parse('1Mb').getValueInBytes()).toBe(1024 * 1024);
+  });
+
   test('throws an error when unsupported unit specified', () => {
     expect(() => ByteSizeValue.parse('1tb')).toThrowErrorMatchingInlineSnapshot(
       `"Failed to parse value as byte value. Value must be either number of bytes, or follow the format <count>[b|kb|mb|gb] (e.g., '1024kb', '200mb', '1gb'), where the number is a safe positive integer."`

packages/kbn-config-schema/src/byte_size_value/index.ts

@@ -22,7 +22,7 @@ function renderUnit(value: number, unit: string) {
 
 export class ByteSizeValue {
   public static parse(text: string): ByteSizeValue {
-    const match = /([1-9][0-9]*)(b|kb|mb|gb)/.exec(text);
+    const match = /([1-9][0-9]*)(b|kb|mb|gb)/i.exec(text);
     if (!match) {
       const number = Number(text);
       if (typeof number !== 'number' || isNaN(number)) {
@@ -35,7 +35,7 @@ export class ByteSizeValue {
     }
 
     const value = parseInt(match[1], 10);
-    const unit = match[2];
+    const unit = match[2].toLowerCase();
 
     return new ByteSizeValue(value * unitMultiplier[unit]);
   }

packages/kbn-dev-utils/src/tooling_log/tooling_log_text_writer.test.ts

@@ -88,3 +88,55 @@ it('formats %s patterns and indents multi-line messages correctly', () => {
   const output = write.mock.calls.reduce((acc, chunk) => `${acc}${chunk}`, '');
   expect(output).toMatchSnapshot();
 });
+
+it('does not write messages from sources in ignoreSources', () => {
+  const write = jest.fn();
+  const writer = new ToolingLogTextWriter({
+    ignoreSources: ['myIgnoredSource'],
+    level: 'debug',
+    writeTo: {
+      write,
+    },
+  });
+
+  writer.write({
+    source: 'myIgnoredSource',
+    type: 'success',
+    indent: 10,
+    args: [
+      '%s\n%O\n\n%d',
+      'foo bar',
+      { foo: { bar: { 1: [1, 2, 3] } }, bar: { bar: { 1: [1, 2, 3] } } },
+      Infinity,
+    ],
+  });
+
+  const output = write.mock.calls.reduce((acc, chunk) => `${acc}${chunk}`, '');
+  expect(output).toEqual('');
+});
+
+it('never ignores write messages from the kibana elasticsearch.deprecation logger context', () => {
+  const write = jest.fn();
+  const writer = new ToolingLogTextWriter({
+    ignoreSources: ['myIgnoredSource'],
+    level: 'debug',
+    writeTo: {
+      write,
+    },
+  });
+
+  writer.write({
+    source: 'myIgnoredSource',
+    type: 'write',
+    indent: 10,
+    args: [
+      '%s\n%O\n\n%d',
+      '[elasticsearch.deprecation]',
+      { foo: { bar: { 1: [1, 2, 3] } }, bar: { bar: { 1: [1, 2, 3] } } },
+      Infinity,
+    ],
+  });
+
+  const output = write.mock.calls.reduce((acc, chunk) => `${acc}${chunk}`, '');
+  expect(output).toMatchSnapshot();
+});

packages/kbn-dev-utils/src/tooling_log/tooling_log_text_writer.ts

@@ -92,7 +92,15 @@ export class ToolingLogTextWriter implements Writer {
     }
 
     if (this.ignoreSources && msg.source && this.ignoreSources.includes(msg.source)) {
-      return false;
+      if (msg.type === 'write') {
+        const txt = format(msg.args[0], ...msg.args.slice(1));
+        // Ensure that Elasticsearch deprecation log messages from Kibana aren't ignored
+        if (!/elasticsearch\.deprecation/.test(txt)) {
+          return false;
+        }
+      } else {
+        return false;
+      }
     }
 
     const prefix = has(MSG_PREFIXES, msg.type) ? MSG_PREFIXES[msg.type] : '';

packages/kbn-es-query/BUILD.bazel

@@ -1,10 +1,11 @@
-load("@npm//@bazel/typescript:index.bzl", "ts_config", "ts_project")
+load("@npm//@bazel/typescript:index.bzl", "ts_config")
 load("@npm//peggy:index.bzl", "peggy")
-load("@build_bazel_rules_nodejs//:index.bzl", "js_library", "pkg_npm")
-load("//src/dev/bazel:index.bzl", "jsts_transpiler")
+load("@build_bazel_rules_nodejs//:index.bzl", "js_library")
+load("//src/dev/bazel:index.bzl", "jsts_transpiler", "pkg_npm", "pkg_npm_types", "ts_project")
 
 PKG_BASE_NAME = "kbn-es-query"
 PKG_REQUIRE_NAME = "@kbn/es-query"
+TYPES_PKG_REQUIRE_NAME = "@types/kbn__es-query"
 
 SOURCE_FILES = glob(
   [
@@ -104,7 +105,7 @@ ts_project(
 js_library(
   name = PKG_BASE_NAME,
   srcs = NPM_MODULE_EXTRA_FILES + [":grammar"],
-  deps = RUNTIME_DEPS + [":target_node", ":target_web", ":tsc_types"],
+  deps = RUNTIME_DEPS + [":target_node", ":target_web"],
   package_name = PKG_REQUIRE_NAME,
   visibility = ["//visibility:public"],
 )
@@ -123,3 +124,20 @@ filegroup(
   ],
   visibility = ["//visibility:public"],
 )
+
+pkg_npm_types(
+  name = "npm_module_types",
+  srcs = SRCS,
+  deps = [":tsc_types"],
+  package_name = TYPES_PKG_REQUIRE_NAME,
+  tsconfig = ":tsconfig",
+  visibility = ["//visibility:public"],
+)
+
+filegroup(
+  name = "build_types",
+  srcs = [
+    ":npm_module_types",
+  ],
+  visibility = ["//visibility:public"],
+)