Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Adds anomaly detection resources and advanced concepts sections #1731

Merged
merged 12 commits into from
Jul 5, 2021
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ To view the `Anomalies` table widget and `Max Anomaly Score By Job` details,
the user must have the `ml_admin` or `ml_user` role.

NOTE: To adjust the `score` threshold that determines which
{ml-docs}/xpack-ml.html[anomalies] are shown, you can modify {kib} ->
{ml-docs}/ml-ad-overview.html[anomalies] are shown, you can modify {kib} ->
Management -> Advanced Settings -> `siem:defaultAnomalyScore`.

[[prebuilt-ml-jobs]]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ In this guide, you’ll learn how to:
Prerequisites:

* This guide assumes you’re already familiar with how to create {anomaly-jobs}.
If not, refer to <<ml-overview>>.
If not, refer to <<ml-ad-overview>>.

The following recommendations are not sequential – the numbers just help to
navigate between the list items; you can take action on one or more of them in
Expand Down
70 changes: 36 additions & 34 deletions docs/en/stack/ml/anomaly-detection/index.asciidoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
include::xpack-ml.asciidoc[]

include::ml-overview.asciidoc[leveloffset=+1]
include::ml-ad-overview.asciidoc[]

include::ml-concepts.asciidoc[leveloffset=+1]

Expand Down Expand Up @@ -28,68 +26,72 @@ include::stopping-ml.asciidoc[leveloffset=+2]

include::ml-restart-failed-jobs.asciidoc[leveloffset=+2]

include::ml-ad-concepts.asciidoc[leveloffset=+1]

include::anomaly-detection-scale.asciidoc[leveloffset=+2]

include::ml-api-quickref.asciidoc[leveloffset=+1]

include::ootb-ml-jobs.asciidoc[leveloffset=+1]
include::anomaly-examples.asciidoc[leveloffset=+1]

include::ootb-ml-jobs-apache.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-alerts.asciidoc[leveloffset=+2]
lcawl marked this conversation as resolved.
Show resolved Hide resolved

include::ootb-ml-jobs-apm.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-aggregations.asciidoc[leveloffset=+2]

include::ootb-ml-jobs-auditbeat.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-detector-custom-rules.asciidoc[leveloffset=+2]

include::ootb-ml-jobs-logs-ui.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-categories.asciidoc[leveloffset=+2]

include::ootb-ml-jobs-metricbeat.asciidoc[leveloffset=+2]
include::geographic-anomalies.asciidoc[leveloffset=+2]

include::ootb-ml-jobs-metrics-ui.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-populations.asciidoc[leveloffset=+2]

include::ootb-ml-jobs-nginx.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-transform.asciidoc[leveloffset=+2]

include::ootb-ml-jobs-siem.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-url.asciidoc[leveloffset=+2]

include::ootb-ml-jobs-uptime.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/ml-delayed-data-detection.asciidoc[leveloffset=+2]

include::{es-repo-dir}/ml/anomaly-detection/functions/ml-functions.asciidoc[leveloffset=+1]
include::mapping-anomalies.asciidoc[leveloffset=+2]

include::{es-repo-dir}/ml/anomaly-detection/functions/ml-count-functions.asciidoc[leveloffset=+2]
include::ml-ad-resources.asciidoc[leveloffset=+1]

include::{es-repo-dir}/ml/anomaly-detection/functions/ml-geo-functions.asciidoc[leveloffset=+2]
include::ml-limitations.asciidoc[leveloffset=+2]

include::{es-repo-dir}/ml/anomaly-detection/functions/ml-info-functions.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/functions/ml-functions.asciidoc[leveloffset=+2]

include::{es-repo-dir}/ml/anomaly-detection/functions/ml-metric-functions.asciidoc[leveloffset=+2]
include::ootb-ml-jobs.asciidoc[leveloffset=+2]

include::{es-repo-dir}/ml/anomaly-detection/functions/ml-rare-functions.asciidoc[leveloffset=+2]
include::ootb-ml-jobs-apache.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/functions/ml-sum-functions.asciidoc[leveloffset=+2]
include::ootb-ml-jobs-apm.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/functions/ml-time-functions.asciidoc[leveloffset=+2]
include::ootb-ml-jobs-auditbeat.asciidoc[]

include::anomaly-examples.asciidoc[leveloffset=+1]
include::ootb-ml-jobs-logs-ui.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-alerts.asciidoc[leveloffset=+2]
include::ootb-ml-jobs-metricbeat.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-aggregations.asciidoc[leveloffset=+2]
include::ootb-ml-jobs-metrics-ui.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-detector-custom-rules.asciidoc[leveloffset=+2]
include::ootb-ml-jobs-nginx.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-categories.asciidoc[leveloffset=+2]
include::ootb-ml-jobs-siem.asciidoc[]

include::geographic-anomalies.asciidoc[leveloffset=+2]
include::ootb-ml-jobs-uptime.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-populations.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/functions/ml-count-functions.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-transform.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/functions/ml-geo-functions.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/ml-configuring-url.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/functions/ml-info-functions.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/ml-delayed-data-detection.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/functions/ml-metric-functions.asciidoc[]

include::mapping-anomalies.asciidoc[leveloffset=+2]
include::{es-repo-dir}/ml/anomaly-detection/functions/ml-rare-functions.asciidoc[]

include::{es-repo-dir}/ml/anomaly-detection/functions/ml-sum-functions.asciidoc[]

include::ml-limitations.asciidoc[leveloffset=+1]
include::{es-repo-dir}/ml/anomaly-detection/functions/ml-time-functions.asciidoc[]

//include::ml-troubleshooting.asciidoc[leveloffset=+1]
//include::ml-troubleshooting.asciidoc[leveloffset=+2]
8 changes: 8 additions & 0 deletions docs/en/stack/ml/anomaly-detection/ml-ad-concepts.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[role="xpack"]
[[ml-ad-concepts]]
= Advanced concepts

This section explains the more complex concepts of the Elastic {ml}
{anomaly-detect} feature.

* <<anomaly-detection-scale>>
Original file line number Diff line number Diff line change
@@ -1,14 +1,24 @@
[role="xpack"]
[[ml-overview]]
= {anomaly-detect-cap} overview
[subs="attributes"]
++++
<titleabbrev>Overview</titleabbrev>
++++
[[ml-ad-overview]]
= {anomaly-detect-cap}

:keywords: {ml-init}, {stack}, {anomaly-detect}, overview
:description: An introduction to {ml} {anomaly-detect}, which analyzes time \
series data to identify and predict anomalous patterns in your data.

Use {anomaly-detect} to analyze time series data by creating accurate baselines
of normal behavior and identifying anomalous patterns in your dataset. Data is
pulled from {es} for analysis and anomaly results are displayed in {kib}
dashboards. Consult <<setup>> to learn more about the licence and the security
privileges that are required to use {anomaly-detect}.

* <<ml-concepts>>
* <<ml-ad-concepts>>
* <<ml-configuration>>
* <<ml-api-quickref>>
* <<anomaly-examples>>
* <<ml-ad-resources>>

[discrete]
[[ml-analyzing]]
== Analyzing the past and present
Expand Down Expand Up @@ -98,5 +108,3 @@ how anomalous new events are.
These results include scores that are aggregated in order to reduce noise and
normalized in order to rank the most mathematically significant anomalies. For
more information, see <<ml-bucket-results>> and <<ml-influencer-results>>.


7 changes: 7 additions & 0 deletions docs/en/stack/ml/anomaly-detection/ml-ad-resources.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[role="xpack"]
[[ml-ad-resources]]
= Resources

This section contains further resources for using {anomaly-detect}.

* <<ml-limitations>>
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
[role="xpack"]
[[ootb-ml-jobs-apache]]
["appendix",role="exclude",id="ootb-ml-jobs-apache"]
= Apache {anomaly-detect} configurations
++++
<titleabbrev>Apache</titleabbrev>
++++

// tag::apache-jobs[]
These {anomaly-job} wizards appear in {kib} if you use the Apache integration in
{fleet} or you use {filebeat} to ship access logs from your
https://httpd.apache.org/[Apache] HTTP servers to {es}. The jobs assume that you
use fields and data types from the Elastic Common Schema (ECS).

[discrete]
[[apache-access-logs]]
== Apache access logs

Expand Down Expand Up @@ -79,6 +77,7 @@ Required {beats} or {agent} integrations:::

* Apache integration

[discrete]
[[apache-access-logs-filebeat]]
== Apache access logs ({filebeat})

Expand Down
10 changes: 4 additions & 6 deletions docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-apm.asciidoc
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@
[role="xpack"]
[[ootb-ml-jobs-apm]]
["appendix",role="exclude",id="ootb-ml-jobs-apm"]
= APM {anomaly-detect} configurations
++++
<titleabbrev>APM</titleabbrev>
++++

These {anomaly-job} wizards appear in {kib} if you have data from APM Agents or
an APM Server stored in {es}. For more details, see the {dfeed} and job
definitions in the `apm_*` folders in
https://github.com/elastic/kibana/tree/{branch}/x-pack/plugins/ml/server/models/data_recognizer/modules[GitHub].

// tag::apm-jobs[]
[discrete]
[[apm-nodejs-jobs]]
== NodeJS
// tag::apm-nodejs-jobs[]
Expand Down Expand Up @@ -40,7 +37,7 @@ than normal.

// end::apm-nodejs-jobs[]


[discrete]
[[apm-rum-javascript-jobs]]
== RUM Javascript
// tag::apm-rum-javascript-jobs[]
Expand Down Expand Up @@ -79,6 +76,7 @@ This job is useful in identifying bots.

// end::apm-rum-javascript-jobs[]

[discrete]
[[apm-transaction-jobs]]
== Transactions
// tag::apm-transaction-jobs[]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
[role="xpack"]
[[ootb-ml-jobs-auditbeat]]
["appendix",role="exclude",id="ootb-ml-jobs-auditbeat"]
= {auditbeat} {anomaly-detect} configurations
++++
<titleabbrev>{auditbeat}</titleabbrev>
++++

// tag::auditbeat-jobs[]
These {anomaly-job} wizards appear in {kib} if you use
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
[role="xpack"]
[[ootb-ml-jobs-logs-ui]]
["appendix",role="exclude",id="ootb-ml-jobs-logs-ui"]
= Logs {anomaly-detect} configurations
++++
<titleabbrev>Logs</titleabbrev>
++++

// tag::logs-jobs[]
These {anomaly-jobs} appear by default in the
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
[role="xpack"]
[[ootb-ml-jobs-metricbeat]]
["appendix",role="exclude",id="ootb-ml-jobs-metricbeat"]
= {metricbeat} {anomaly-detect} configurations
++++
<titleabbrev>{metricbeat}</titleabbrev>
++++

// tag::metricbeat-jobs[]
These {anomaly-job} wizards appear in {kib} if you use the
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
[role="xpack"]
[[ootb-ml-jobs-metrics-ui]]
["appendix",role="exclude",id="ootb-ml-jobs-metrics-ui"]
= Metrics {anomaly-detect} configurations
++++
<titleabbrev>Metrics</titleabbrev>
++++

// tag::metrics-jobs[]
These {anomaly-jobs} can be created in the
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
[role="xpack"]
[[ootb-ml-jobs-nginx]]
["appendix",role="exclude",id="ootb-ml-jobs-nginx"]
= Nginx {anomaly-detect} configurations
++++
<titleabbrev>Nginx</titleabbrev>
++++

// tag::nginx-jobs[]

Expand All @@ -12,6 +8,7 @@ These {anomaly-job} wizards appear in {kib} if you use the Nginx integration in
http://nginx.org/[Nginx] HTTP servers to {es}. The jobs assume that you use
fields and data types from the Elastic Common Schema (ECS).

[discrete]
[[nginx-access-logs]]
== Nginx access logs

Expand Down Expand Up @@ -81,6 +78,7 @@ Required {beats} or {agent} integrations:::

* Nginx integration

[discrete]
[[nginx-access-logs-filebeat]]
== Nginx access logs ({filebeat})

Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
[role="xpack"]
[[ootb-ml-jobs-siem]]
["appendix",role="exclude",id="ootb-ml-jobs-siem"]
= Security {anomaly-detect} configurations
++++
<titleabbrev>Security</titleabbrev>
++++

// tag::siem-jobs[]
These {anomaly-jobs} automatically detect file system and network anomalies on
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
[role="xpack"]
[[ootb-ml-jobs-uptime]]
["appendix",role="exclude",id="ootb-ml-jobs-uptime"]
= Uptime {anomaly-detect} configurations
++++
<titleabbrev>Uptime</titleabbrev>
++++

// tag::uptime-jobs[]

If you have appropriate {heartbeat} data in {es}, you can enable this
Expand Down
18 changes: 9 additions & 9 deletions docs/en/stack/ml/anomaly-detection/ootb-ml-jobs.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,15 @@ Refer to <<create-jobs>> to learn more about creating a job by using supplied
configurations. Logs and Metrics supplied configurations are available and can
be created via the related solution UI in {kib}.

* <<ootb-ml-jobs-apache>>
* <<ootb-ml-jobs-apm>>
* <<ootb-ml-jobs-auditbeat>>
* <<ootb-ml-jobs-logs-ui>>
* <<ootb-ml-jobs-metricbeat>>
* <<ootb-ml-jobs-metrics-ui>>
* <<ootb-ml-jobs-nginx>>
* <<ootb-ml-jobs-siem>>
* <<ootb-ml-jobs-uptime>>
* <<ootb-ml-jobs-apache,Apache>>
* <<ootb-ml-jobs-apm,APM>>
* <<ootb-ml-jobs-auditbeat,{auditbeat}>>
* <<ootb-ml-jobs-logs-ui,Logs>>
* <<ootb-ml-jobs-metricbeat,{metricbeat}>>
* <<ootb-ml-jobs-metrics-ui,Metrics>>
* <<ootb-ml-jobs-nginx,Nginx>>
* <<ootb-ml-jobs-siem,Security>>
* <<ootb-ml-jobs-uptime,Uptiime>>


NOTE: The configurations are only available if data exists that matches the
Expand Down
23 changes: 0 additions & 23 deletions docs/en/stack/ml/anomaly-detection/xpack-ml.asciidoc

This file was deleted.

Loading