Merge branch 'master' into feature/issue-49871--added-options-in-embe…

…ddable-maps

docs/logs/index.asciidoc

@@ -27,3 +27,5 @@ include::getting-started.asciidoc[]
 include::using.asciidoc[]
 
 include::configuring.asciidoc[]
+
+include::log-rate.asciidoc[]

docs/logs/log-rate.asciidoc

@@ -0,0 +1,94 @@
+[role="xpack"]
+[[xpack-logs-analysis]]
+== Detecting and inspecting log anomalies
+
+beta::[]
+
+When the {ml} {anomaly-detect} features are enabled,
+you can use the **Log rate** page in the Logs app.
+**Log rate** helps you to detect and inspect log anomalies and the log partitions where the log anomalies occur.
+This means you can easily spot anomalous behavior without significant human intervention --
+no more manually sampling log data, calculating rates, and determining if rates are normal.
+
+*Log rate* automatically highlights periods of time where the log rate is outside expected bounds,
+and therefore may be anomalous.
+You can use this information as a basis for further investigations.
+For example:
+
+* A significant drop in the log rate might suggest that a piece of infrastructure stopped responding,
+and thus we're serving less requests.
+* A spike in the log rate could denote a DDoS attack.
+This may lead to an investigation of IP addresses from incoming requests.
+
+You can also view log anomalies directly in the <<xpack-ml-anomalies,Machine Learning app>>.
+
+[float]
+[[logs-analysis-create-ml-job]]
+=== Enable log rate analysis and anomaly detection
+
+Create a machine learning job to enable log rate analysis and anomaly detection.
+
+[role="screenshot"]
+image::logs/images/analysis-tab-create-ml-job.png[Create machine learning job]
+
+1. To enable log rate analysis and anomaly detection,
+you must first create your own {kibana-ref}/xpack-spaces.html[space].
+2. Within a space, navigate to the Logs app and select *Log rate*.
+Here, you'll be prompted to create a machine learning job which will carry out the log rate analysis.
+3. Choose a time range for the machine learning analysis.
+4. Add the Indices that contain the logs you want to analyze.
+5. Click *Create ML job*.
+6. You're now ready to analyze your log partitions.
+
+Even though the machine learning job's time range is fixed,
+you can still use the time filter to adjust the results that are shown in your analysis.
+
+[role="screenshot"]
+image::logs/images/log-time-filter.png[Log rate time filter]
+
+[float]
+[[logs-analysis-entries-chart]]
+=== Log entries chart
+
+The log entries chart shows an overall, color-coded visualization of the log entry rate,
+partitioned according to the value of the Elastic Common Schema (ECS)
+{ecs-ref}/ecs-event.html[`event.dataset`] field.
+This chart helps you quickly spot increases or decreases in each partition's log rate.
+
+[role="screenshot"]
+image::logs/images/log-rate-entries.png[Log rate entries chart]
+
+If you have a lot of log partitions, use the following to filter your data:
+
+* Hover over a time range to see the log rate for each partition.
+* Click or hover on a partition name to show, hide, or highlight the partition values.
+
+[float]
+[[logs-analysis-anomalies-chart]]
+=== Anomalies charts
+
+The Anomalies chart shows the time range where anomalies were detected.
+The typical rate values are shown in grey, while the anomalous regions are color-coded and superimposed on top.
+
+[role="screenshot"]
+image::logs/images/log-rate-anomalies.png[Log rate entries chart]
+
+When a time range is flagged as anomalous,
+the machine learning algorithms have detected unusual log rate activity.
+This might be because:
+
+* The log rate is significantly higher than usual.
+* The log rate is significantly lower than usual.
+* Other anomalous behavior has been detected.
+For example, the log rate is within bounds, but not fluctuating when it is expected to.
+
+The level of anomaly detected in a time period is color-coded, from red, orange, yellow, to blue.
+Red indicates a critical anomaly level, while blue is a warning level.
+
+To help you further drill down into a potential anomaly,
+you can view an anomaly chart for each individual partition:
+
+Anomaly scores range from 0 (no anomalies) to 100 (critical).
+
+To analyze the anomalies in more detail, click *Analyze in ML*, which opens the
+{kibana-ref}/xpack-ml.html[Anomaly Explorer in Machine Learning].

docs/logs/using.asciidoc

@@ -78,8 +78,19 @@ This opens the *Log event document details* fly-out that shows the fields associ
 To quickly filter the logs stream by one of the field values, in the log event details, click the *View event with filter* icon image:logs/images/logs-view-event-with-filter.png[View event icon] beside the field.
 This automatically adds a search filter to the logs stream to filter the entries by this field and value.
 
-To see other actions related to the event, in the log event details, click *Actions*.
-Depending on the event and the features you have installed and configured, you may also be able to:
+[float]
+[[view-log-anomalies]]
+=== View log anomalies
+
+When the machine learning anomaly detection features are enabled, click *Log rate*, which allows you to
+<<xpack-logs-analysis,use machine learning to detect and inspect anomalies>> in your log data.
+
+[float]
+[[logs-integrations]]
+=== Logs app integrations
+
+To see other actions related to the event, click *Actions* in the log event details.
+Depending on the event and the features you have configured, you may also be able to:
 
 * Select *View status in Uptime* to <<uptime-overview, view related uptime information>> in the *Uptime* app.
 * Select *View in APM* to <<traces, view related APM traces>> in the *APM* app.

docs/settings/security-settings.asciidoc

@@ -40,6 +40,8 @@ An arbitrary string of 32 characters or more that is used to encrypt credentials
 in a cookie. It is crucial that this key is not exposed to users of {kib}. By
 default, a value is automatically generated in memory. If you use that default
 behavior, all sessions are invalidated when {kib} restarts.
+In addition, high-availability deployments of {kib} will behave unexpectedly 
+if this setting isn't the same for all instances of {kib}.
 
 `xpack.security.secureCookies`::
 Sets the `secure` flag of the session cookie. The default value is `false`. It

src/legacy/core_plugins/apm_oss/index.js

@@ -38,7 +38,6 @@ export default function apmOss(kibana) {
         spanIndices: Joi.string().default('apm-*'),
         metricsIndices: Joi.string().default('apm-*'),
         onboardingIndices: Joi.string().default('apm-*'),
-        apmAgentConfigurationIndex: Joi.string().default('.apm-agent-configuration'),
       }).default();
     },
 

src/legacy/core_plugins/data/public/filter/apply_filters/apply_filter_popover_content.tsx

@@ -32,8 +32,7 @@ import { FormattedMessage } from '@kbn/i18n/react';
 import React, { Component } from 'react';
 import { IndexPattern } from '../../index_patterns';
 import { FilterLabel } from '../filter_bar/filter_editor/lib/filter_label';
-import { mapAndFlattenFilters, esFilters } from '../../../../../../plugins/data/public';
-import { getDisplayValueFromFilter } from '../filter_bar/filter_editor/lib/get_display_value';
+import { mapAndFlattenFilters, esFilters, utils } from '../../../../../../plugins/data/public';
 
 interface Props {
   filters: esFilters.Filter[];
@@ -58,7 +57,7 @@ export class ApplyFiltersPopoverContent extends Component<Props, State> {
     };
   }
   private getLabel(filter: esFilters.Filter) {
-    const valueLabel = getDisplayValueFromFilter(filter, this.props.indexPatterns);
+    const valueLabel = utils.getDisplayValueFromFilter(filter, this.props.indexPatterns);
     return <FilterLabel filter={filter} valueLabel={valueLabel} />;
   }
 

src/legacy/core_plugins/data/public/filter/filter_bar/filter_bar.tsx

@@ -21,18 +21,18 @@ import { EuiButtonEmpty, EuiFlexGroup, EuiFlexItem, EuiPopover } from '@elastic/
 import { FormattedMessage, InjectedIntl, injectI18n } from '@kbn/i18n/react';
 import classNames from 'classnames';
 import React, { useState } from 'react';
-import { IndexPattern } from '../../index_patterns';
+
 import { FilterEditor } from './filter_editor';
 import { FilterItem } from './filter_item';
 import { FilterOptions } from './filter_options';
 import { useKibana } from '../../../../../../plugins/kibana_react/public';
-import { esFilters } from '../../../../../../plugins/data/public';
+import { IIndexPattern, esFilters } from '../../../../../../plugins/data/public';
 
 interface Props {
   filters: esFilters.Filter[];
   onFiltersUpdated?: (filters: esFilters.Filter[]) => void;
   className: string;
-  indexPatterns: IndexPattern[];
+  indexPatterns: IIndexPattern[];
   intl: InjectedIntl;
 }
 

src/legacy/core_plugins/data/public/filter/filter_bar/filter_editor/index.tsx

@@ -36,37 +36,36 @@ import { i18n } from '@kbn/i18n';
 import { FormattedMessage, InjectedIntl, injectI18n } from '@kbn/i18n/react';
 import { get } from 'lodash';
 import React, { Component } from 'react';
-import { Field, IndexPattern } from '../../../index_patterns';
 import { GenericComboBox, GenericComboBoxProps } from './generic_combo_box';
 import {
-  buildCustomFilter,
-  buildFilter,
   getFieldFromFilter,
   getFilterableFields,
-  getFilterParams,
-  getIndexPatternFromFilter,
   getOperatorFromFilter,
   getOperatorOptions,
-  getQueryDslFromFilter,
   isFilterValid,
 } from './lib/filter_editor_utils';
 import { Operator } from './lib/filter_operators';
 import { PhraseValueInput } from './phrase_value_input';
 import { PhrasesValuesInput } from './phrases_values_input';
 import { RangeValueInput } from './range_value_input';
-import { esFilters } from '../../../../../../../plugins/data/public';
+import {
+  esFilters,
+  utils,
+  IIndexPattern,
+  IFieldType,
+} from '../../../../../../../plugins/data/public';
 
 interface Props {
   filter: esFilters.Filter;
-  indexPatterns: IndexPattern[];
+  indexPatterns: IIndexPattern[];
   onSubmit: (filter: esFilters.Filter) => void;
   onCancel: () => void;
   intl: InjectedIntl;
 }
 
 interface State {
-  selectedIndexPattern?: IndexPattern;
-  selectedField?: Field;
+  selectedIndexPattern?: IIndexPattern;
+  selectedField?: IFieldType;
   selectedOperator?: Operator;
   params: any;
   useCustomLabel: boolean;
@@ -82,10 +81,10 @@ class FilterEditorUI extends Component<Props, State> {
       selectedIndexPattern: this.getIndexPatternFromFilter(),
       selectedField: this.getFieldFromFilter(),
       selectedOperator: this.getSelectedOperator(),
-      params: getFilterParams(props.filter),
+      params: esFilters.getFilterParams(props.filter),
       useCustomLabel: props.filter.meta.alias !== null,
       customLabel: props.filter.meta.alias,
-      queryDsl: JSON.stringify(getQueryDslFromFilter(props.filter), null, 2),
+      queryDsl: JSON.stringify(esFilters.cleanFilter(props.filter), null, 2),
       isCustomEditorOpen: this.isUnknownFilterType(),
     };
   }
@@ -377,7 +376,7 @@ class FilterEditorUI extends Component<Props, State> {
   }
 
   private getIndexPatternFromFilter() {
-    return getIndexPatternFromFilter(this.props.filter, this.props.indexPatterns);
+    return utils.getIndexPatternFromFilter(this.props.filter, this.props.indexPatterns);
   }
 
   private getFieldFromFilter() {
@@ -412,14 +411,14 @@ class FilterEditorUI extends Component<Props, State> {
     return isFilterValid(indexPattern, field, operator, params);
   }
 
-  private onIndexPatternChange = ([selectedIndexPattern]: IndexPattern[]) => {
+  private onIndexPatternChange = ([selectedIndexPattern]: IIndexPattern[]) => {
     const selectedField = undefined;
     const selectedOperator = undefined;
     const params = undefined;
     this.setState({ selectedIndexPattern, selectedField, selectedOperator, params });
   };
 
-  private onFieldChange = ([selectedField]: Field[]) => {
+  private onFieldChange = ([selectedField]: IFieldType[]) => {
     const selectedOperator = undefined;
     const params = undefined;
     this.setState({ selectedField, selectedOperator, params });
@@ -475,13 +474,21 @@ class FilterEditorUI extends Component<Props, State> {
       const { index, disabled, negate } = this.props.filter.meta;
       const newIndex = index || this.props.indexPatterns[0].id!;
       const body = JSON.parse(queryDsl);
-      const filter = buildCustomFilter(newIndex, body, disabled, negate, alias, $state.store);
+      const filter = esFilters.buildCustomFilter(
+        newIndex,
+        body,
+        disabled,
+        negate,
+        alias,
+        $state.store
+      );
       this.props.onSubmit(filter);
     } else if (indexPattern && field && operator) {
-      const filter = buildFilter(
+      const filter = esFilters.buildFilter(
         indexPattern,
         field,
-        operator,
+        operator.type,
+        operator.negate,
         this.props.filter.meta.disabled,
         params,
         alias,
@@ -492,11 +499,11 @@ class FilterEditorUI extends Component<Props, State> {
   };
 }
 
-function IndexPatternComboBox(props: GenericComboBoxProps<IndexPattern>) {
+function IndexPatternComboBox(props: GenericComboBoxProps<IIndexPattern>) {
   return GenericComboBox(props);
 }
 
-function FieldComboBox(props: GenericComboBoxProps<Field>) {
+function FieldComboBox(props: GenericComboBoxProps<IFieldType>) {
   return GenericComboBox(props);
 }