[Logs UI] Create ML module for log analysis

[weltenwort]

Summary

This adds an ML module called logs_ui_analysis, which contains a job definition to detect anomalies in the log rate.

⚠️ Some fields are designed to be overridden at setup time using #42946:

• in the job log-entry-rate
  • analysis_config.bucket_span
  • data_description.time_field
• in the datafeed datafeed-log-entry-rate:
  • aggregations.buckets.date_histogram.field
  • aggregations.buckets.date_histogram.fixed_interval
  • aggregations.buckets.aggregations[timestampField]
  • aggregations.buckets.aggregations[timestampField].max.field
  • aggregations.buckets.aggregations.doc_count_per_minute.bucket_script.script.params.bucket_span_in_ms

closes #42593

Implementation notes

• The datafeed scales the doc_count to one minute using a bucket_script aggregation called doc_count_per_minute. That aggregation is used as the summary_count_field_name in the job configuration. The scaling has several advantages:
  • Denominators of 1 in the unit reduce the risk of misinterpretation (doc_count / minute compared to doc_count / (15 minutes).
  • Future changes of bucket span sizes are simpler because the result's unit is not as tightly coupled to it.
• The max timestamp aggregation is not included in the datafeed, because its name depends on the time field name. Since the additive nature of the overrides makes a removal of the predefined aggregation impossible, it needs to be specified solely as an override at setup time.

Testing hints

The module can be deployed via a call to Kibana's ML module setup api at /api/ml/modules/setup/logs_ui_analysis. The POST requests body needs to specify a few parameters, e.g.:

{
  "indexPatternName": "filebeat-*",
  "prefix": "kibana-logs-ui-testspace-default-",
  "startDatafeed": true,
  "jobOverrides": [
    {
      "job_id": "log-entry-rate",
      "analysis_config": {
        "bucket_span": "900000ms"
      },
      "data_description": {
        "time_field": "@timestamp"
      }
    }
  ],
  "datafeedOverrides": [
    {
      "job_id": "log-entry-rate",
      "aggregations": {
        "buckets": {
          "date_histogram": {
            "field": "@timestamp",
            "fixed_interval": "900000ms"
          },
          "aggregations": {
            "@timestamp": {
              "max": {
                "field": "@timestamp"
              }
            },
            "doc_count_per_minute": {
              "bucket_script": {
                "script": {
                  "params": {
                    "bucket_span_in_ms": 900000
                  }
                }
              }
            }
          }
        }
      }
    }
  ]
}

[elasticmachine]

Pinging @elastic/infra-logs-ui

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[jasonrhodes]

@elastic/ml-ui any feedback you all have or anything else you need from us on this? We are beginning to build the UI that connects to this endpoint and are hoping to test it soon. Thanks!

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[walterra]

Using the testing hints I was able to create a job based on filebeat data, so the setup in general LGTM.

Looks good in Single Metric Viewer:

(I'm not familiar with the use case so I cannot really assess the quality of the job/datafeed config, but code LGTM)

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request