Merge branch 'main' of https://github.com/elastic/kibana into move-st…

…ructure-alerts-api

docs/CHANGELOG.asciidoc

@@ -194,8 +194,8 @@ Discover::
 * Allow fetching more documents on Discover page ({kibana-pull}163784[#163784]).
 Elastic Security::
 For the Elastic Security 8.11.0 release information, refer to {security-guide}/release-notes.html[_Elastic Security Solution Release Notes_].
-Enterprise Search::
-For the Elastic Enterprise Search 8.11.0 release information, refer to {enterprise-search-ref}/changelog.html[_Elastic Enterprise Search Documentation Release notes_].
+Search::
+* Self-managed connector clients now show advanced configuration options in the UI ({kibana-pull}167770[#167770]).
 Fleet::
 * Adds sidebar navigation showing headings extracted from the readme ({kibana-pull}167216[#167216]).
 Inspector::
@@ -252,8 +252,15 @@ Dashboard::
 * Generate new panel IDs on Dashboard clone ({kibana-pull}166299[#166299]).
 Elastic Security::
 For the Elastic Security 8.11.0 release information, refer to {security-guide}/release-notes.html[_Elastic Security Solution Release Notes_].
-Enterprise Search::
-For the Elastic Enterprise Search 8.11.0 release information, refer to {enterprise-search-ref}/changelog.html[_Elastic Enterprise Search Documentation Release notes_].
+Search::
+* Native connector external documentation links are now rendered conditionally to avoid empty links ({kibana-pull}169121[#169121]).
+* Fixed an issue which caused Access Control Syncs to be scheduled when Document Level Security was disabled ({kibana-pull}168987[#168987]).
+* Restored access and admin checks for App Search and Workplace Search product cards ({kibana-pull}168890[#168890]).
+* The filter box in the *Browse documents* tab under *Search > Content > Indices* now escapes Lucene reserved characters instead of throwing errors ({kibana-pull}168092[#168092]).
+* Fixed an issue associated with changing the indices underlying a search application. When a user modifies the indices underlying a search application in Kibana, the associated search template is now reverted to the default template ({kibana-pull}167532[#167532]).
+* Fixed an issue where the Search plugin was inaccessible for unauthenticated users, eg. for Kibana in read-only demo setups ({kibana-pull}167171[#167171]).
+* Fixed an issue with the welcome banner in Search ({kibana-pull}166814[#166814]).
+* Self managed connector clients now show advanced configuration options in the UI ({kibana-pull}167770[#167770]).
 Fleet::
 * Vastly improve performance of Fleet final pipeline's date formatting logic for `event.ingested` ({kibana-pull}167318[#167318]).
 Lens & Visualizations::

docs/concepts/esql.asciidoc

@@ -0,0 +1,40 @@
+[[esql]]
+=== {esql}
+
+preview::[]
+
+The Elasticsearch Query Language, {esql}, has been created to make exploring your data faster and easier using the **Discover** application. From version 8.11 you can try this new feature, which is enabled by default. 
+
+[role="screenshot"]
+image:images/esql-data-view-menu.png[An image of the Discover UI where users can access the {esql} feature, width=30%]
+
+This new piped language allows you to chain together multiple commands to query your data. Based on the query, Lens suggestions in Discover create a visualization of the query results.
+
+{esql} comes with its own dedicated {esql} Compute Engine for greater efficiency. From one query you can search, aggregate, calculate and perform data transformations without leaving **Discover**. Write your query directly in **Discover** or use the **Dev Tools** with the {ref}/esql-rest.html[{esql} API]. 
+
+{esql} also features in-app help, so you can get started faster and don't have to leave the application to check syntax. 
+
+[role="screenshot"]
+image:images/esql-in-app-help.png[An image of the Discover UI where users can browse the in-app help]
+
+For more detailed information about the {esql} language, refer to {ref}/esql-language.html[Learning {esql}].
+
+[float]
+[[esql-observability]]
+==== {observability}
+
+{esql} makes it much easier to analyze metrics, logs and traces from a single query. Find performance issues fast by defining fields on the fly, enriching data with lookups, and using simultaneous query processing. Combining {esql} with {ml} and AiOps can improve detection accuracy and use aggregated value thresholds.   
+
+[float]
+[[esql-security]]
+==== Security 
+
+Use {esql} to retrieve important information for investigation by using lookups. Enrich data and create new fields on the go to gain valuable insight for faster decision-making and actions. For example, perform a lookup on an IP address to identify its geographical location, its association with known malicious entities, or whether it belongs to a known cloud service provider all from one search bar. {esql} ensures more accurate alerts by incorporating aggregated values in detection rules.
+
+[float]
+[[esql-whats-next]]
+==== What's next?
+
+Full documentation for this language is available in the {es} documentation, refer to {ref}/esql.html[{esql}].
+
+Alternatively, a short tutorial is available in the **Discover** section <<try-esql, Try {esql}L>>.

docs/concepts/index.asciidoc

@@ -155,8 +155,11 @@ include::data-views.asciidoc[]
 
 include::set-time-filter.asciidoc[]
 
+include::esql.asciidoc[]
+
 include::kuery.asciidoc[]
 
 include::lucene.asciidoc[]
 
 include::save-query.asciidoc[]
+

docs/discover/try-esql.asciidoc

@@ -0,0 +1,91 @@
+[[try-esql]]
+== Try {esql}
+
+preview::[]
+
+The Elasticsearch Query Language, {esql}, makes it easier to explore your data without leaving Discover. 
+
+In this tutorial we'll use the {kib} sample web logs in Discover and Lens to explore the data and create visualizations. 
+
+[float]
+[[prerequisite]]
+=== Prerequisite 
+
+To be able to select **Try {esql}** from the Data views menu the `discover:enableESQL` setting must be enabled from **Stack Management > Advanced Settings**. It is enabled by default. 
+
+[float]
+[[tutorial-try-esql]]
+=== Trying {esql}
+
+To load the sample data:
+
+. On the home page, click **Try sample data**.
+. Click **Other sample data sets**.
+. On the Sample web logs card, click **Add data**.
+. Open the main menu and select *Discover*.
+. From the Data views menu, select *Try {esql}*.
+
+Let's say we want to find out what operating system users have and how much RAM is on their machine.  
+
+. Set the time range to **Last 7 days**.
+. Expand image:images/expand-icon-2.png[An image of the expand icon] the query bar.
+. Put each processing command on a new line for better readability.
+. Copy the query below:
++
+[source,esql]
+----
+FROM kibana_sample_data_logs 
+| KEEP machine.os, machine.ram
+----
++
+. Click **Update**.
++
+[role="screenshot"]
+image:images/esql-machine-os-ram.png[An image of the query result]
++
+[NOTE]
+====
+{esql} keywords are not case sensitive. 
+====
+
+Let's add `geo.dest` to our query, to find out the geographical destination of the visits, and limit the results. 
+
+. Copy the query below:
++
+[source,esql]
+----
+FROM kibana_sample_data_logs 
+| KEEP machine.os, machine.ram, geo.dest
+| LIMIT 10
+----
++
+. Click **Update**.
++
+[role="screenshot"]
+image:images/esql-limit.png[An image of the extended query result]
+
+Let's sort the data by machine ram and filter out the destination GB. 
+
+. Copy the query below:
++
+[source,esql]
+----
+FROM kibana_sample_data_logs 
+| KEEP machine.os, machine.ram, geo.dest
+| SORT machine.ram desc
+| WHERE geo.dest != "GB"
+| LIMIT 10
+----
++
+. Click **Update**.
++
+[role="screenshot"]
+image:images/esql-full-query.png[]
++
+. Click **Save** to save the query and visualization to a dashboard. 
+
+To make changes to the visualization you can use the visualization drop-down. To make changes to the colors used or the axes, or click the pencil icon. This opens an in-line editor where you can change the colors and axes of the visualization. 
+
+To learn more about {esql}, try other tutorials, see more examples and reference material, refer to {ref}/esql.html[{esql}].
+
+

docs/user/discover.asciidoc

@@ -346,4 +346,7 @@ include::{kib-repo-dir}/discover/field-statistics.asciidoc[]
 
 include::{kib-repo-dir}/discover/log-pattern-analysis.asciidoc[]
 
-include::{kib-repo-dir}/discover/search-sessions.asciidoc[]
+include::{kib-repo-dir}/discover/search-sessions.asciidoc[]
+
+include::{kib-repo-dir}/discover/try-esql.asciidoc[]
+

package.json

@@ -1459,7 +1459,7 @@
     "blob-polyfill": "^7.0.20220408",
     "callsites": "^3.1.0",
     "chance": "1.0.18",
-    "chromedriver": "^117.0.3",
+    "chromedriver": "^119.0.0",
     "clean-webpack-plugin": "^3.0.0",
     "cli-table3": "^0.6.1",
     "compression-webpack-plugin": "^4.0.0",

packages/core/http/core-http-browser-internal/src/fetch.ts

@@ -23,6 +23,7 @@ import {
   ELASTIC_HTTP_VERSION_HEADER,
   X_ELASTIC_INTERNAL_ORIGIN_REQUEST,
 } from '@kbn/core-http-common';
+import { KIBANA_BUILD_NR_HEADER } from '@kbn/core-http-common';
 import { HttpFetchError } from './http_fetch_error';
 import { HttpInterceptController } from './http_intercept_controller';
 import { interceptRequest, interceptResponse } from './intercept';
@@ -136,7 +137,7 @@ export class Fetch {
         'Content-Type': 'application/json',
         ...options.headers,
         'kbn-version': this.params.kibanaVersion,
-        'kbn-build-number': this.params.buildNumber,
+        [KIBANA_BUILD_NR_HEADER]: this.params.buildNumber,
         [ELASTIC_HTTP_VERSION_HEADER]: version,
         [X_ELASTIC_INTERNAL_ORIGIN_REQUEST]: 'Kibana',
         ...(!isEmpty(context) ? new ExecutionContextContainer(context).toHeader() : {}),

packages/core/http/core-http-common/index.ts

@@ -14,4 +14,5 @@ export {
   ELASTIC_HTTP_VERSION_QUERY_PARAM,
   ELASTIC_INTERNAL_ORIGIN_QUERY_PARAM,
   X_ELASTIC_INTERNAL_ORIGIN_REQUEST,
+  KIBANA_BUILD_NR_HEADER,
 } from './src/constants';

packages/core/http/core-http-common/src/constants.ts

@@ -11,3 +11,6 @@ export const ELASTIC_HTTP_VERSION_HEADER = 'elastic-api-version' as const;
 export const ELASTIC_HTTP_VERSION_QUERY_PARAM = 'apiVersion' as const;
 export const ELASTIC_INTERNAL_ORIGIN_QUERY_PARAM = 'elasticInternalOrigin' as const;
 export const X_ELASTIC_INTERNAL_ORIGIN_REQUEST = 'x-elastic-internal-origin' as const;
+
+/** @internal */
+export const KIBANA_BUILD_NR_HEADER = 'kbn-build-number' as const;

packages/core/http/core-http-server-internal/src/http_service.ts

@@ -102,7 +102,7 @@ export class HttpService
       },
     });
 
-    registerCoreHandlers(prebootSetup, config, this.env);
+    registerCoreHandlers(prebootSetup, config, this.env, this.log);
 
     if (this.shouldListen(config)) {
       this.log.debug('starting preboot server');
@@ -162,7 +162,7 @@ export class HttpService
       deps.executionContext
     );
 
-    registerCoreHandlers(serverContract, config, this.env);
+    registerCoreHandlers(serverContract, config, this.env, this.log);
 
     this.internalSetup = {
       ...serverContract,

packages/core/http/core-http-server-internal/src/lifecycle_handlers.test.ts

@@ -14,16 +14,21 @@ import type {
   OnPostAuthToolkit,
   OnPreRoutingToolkit,
   OnPostAuthHandler,
+  OnPreResponseInfo,
 } from '@kbn/core-http-server';
 import { mockRouter } from '@kbn/core-http-router-server-mocks';
 import {
+  createBuildNrMismatchLoggerPreResponseHandler,
   createCustomHeadersPreResponseHandler,
   createRestrictInternalRoutesPostAuthHandler,
   createVersionCheckPostAuthHandler,
   createXsrfPostAuthHandler,
 } from './lifecycle_handlers';
 
 import { HttpConfig } from './http_config';
+import { loggerMock } from '@kbn/logging-mocks';
+import { Logger } from '@kbn/logging';
+import { KIBANA_BUILD_NR_HEADER } from '@kbn/core-http-common';
 
 type ToolkitMock = jest.Mocked<OnPreResponseToolkit & OnPostAuthToolkit & OnPreRoutingToolkit>;
 
@@ -43,13 +48,18 @@ const forgeRequest = ({
   path = '/',
   method = 'get',
   kibanaRouteOptions,
+  buildNr,
 }: Partial<{
   headers: Record<string, string>;
   query: Record<string, string>;
   path: string;
   method: RouteMethod;
   kibanaRouteOptions: KibanaRouteOptions;
+  buildNr: undefined | string;
 }>): KibanaRequest => {
+  if (buildNr) {
+    headers[KIBANA_BUILD_NR_HEADER] = buildNr;
+  }
   return mockRouter.createKibanaRequest({
     headers,
     path,
@@ -452,3 +462,79 @@ describe('customHeaders pre-response handler', () => {
     });
   });
 });
+
+describe('build number mismatch logger on error pre-response handler', () => {
+  let logger: jest.Mocked<Logger>;
+
+  beforeEach(() => {
+    logger = loggerMock.create();
+  });
+
+  it('injects a logger prefix', () => {
+    createBuildNrMismatchLoggerPreResponseHandler(123, logger);
+    expect(logger.get).toHaveBeenCalledTimes(1);
+    expect(logger.get).toHaveBeenCalledWith(`kbn-build-number-mismatch`);
+  });
+
+  it('does not log for same server-client build', () => {
+    const handler = createBuildNrMismatchLoggerPreResponseHandler(123, logger);
+    const request = forgeRequest({ buildNr: '123' });
+    const response: OnPreResponseInfo = { statusCode: 500 }; // should log for errors, but not this time bc same build nr
+    handler(request, response, createToolkit());
+    expect(logger.warn).not.toHaveBeenCalled();
+  });
+
+  const badStatusCodeTestCases = [
+    /** just test a few common ones */
+    [400],
+    [401],
+    [403],
+    [499],
+    [500],
+    [502],
+    [999] /* and not so common... */,
+  ];
+  it.each(badStatusCodeTestCases)(
+    'logs for %p responses and newer client builds',
+    (responseStatusCode) => {
+      const handler = createBuildNrMismatchLoggerPreResponseHandler(123, logger);
+      const request = forgeRequest({ buildNr: '124' });
+      const response: OnPreResponseInfo = { statusCode: responseStatusCode };
+      handler(request, response, createToolkit());
+      expect(logger.warn).toHaveBeenCalledTimes(1);
+      expect(logger.warn).toHaveBeenCalledWith(
+        `Client build (124) is newer than this Kibana server build (123). The [${responseStatusCode}] error status in req id [123] may be due to client-server incompatibility!`
+      );
+    }
+  );
+
+  it.each(badStatusCodeTestCases)('logs for %p responses and older client builds', (statusCode) => {
+    const handler = createBuildNrMismatchLoggerPreResponseHandler(123, logger);
+    const request = forgeRequest({ buildNr: '122' });
+    const response: OnPreResponseInfo = { statusCode };
+    handler(request, response, createToolkit());
+    expect(logger.warn).toHaveBeenCalledTimes(1);
+    expect(logger.warn).toHaveBeenCalledWith(
+      `Client build (122) is older than this Kibana server build (123). The [${statusCode}] error status in req id [123] may be due to client-server incompatibility!`
+    );
+  });
+
+  it.each([[200], [201], [301], [302]])('does not log for %p responses', (statusCode) => {
+    const handler = createBuildNrMismatchLoggerPreResponseHandler(123, logger);
+    const request = forgeRequest({ buildNr: '124' });
+    const response: OnPreResponseInfo = { statusCode };
+    handler(request, response, createToolkit());
+    expect(logger.warn).not.toHaveBeenCalled();
+  });
+
+  it.each([['foo'], [['yes']], [true], [null], [[]], [undefined]])(
+    'ignores bogus client build numbers like %p',
+    (bogusBuild) => {
+      const handler = createBuildNrMismatchLoggerPreResponseHandler(123, logger);
+      const request = forgeRequest({ buildNr: bogusBuild as any });
+      const response: OnPreResponseInfo = { statusCode: 500 };
+      handler(request, response, createToolkit());
+      expect(logger.warn).not.toHaveBeenCalled();
+    }
+  );
+});