Merge branch 'master' of github.com:elastic/kibana into issue-106980

.gitignore

@@ -5,6 +5,7 @@
 /.chromium
 .DS_Store
 .node_binaries
+/.beats
 .native_modules
 node_modules
 !/src/dev/npm/integration_tests/__fixtures__/fixture1/node_modules

dev_docs/troubleshooting.mdx

@@ -0,0 +1,28 @@
+---
+id: kibTroubleshooting
+slug: /kibana-dev-docs/troubleshooting
+title: Troubleshooting
+summary: A collection of tips for working around strange issues.
+date: 2021-09-08
+tags: ['kibana', 'onboarding', 'dev', 'troubleshooting']
+---
+
+### Typescript issues
+
+When switching branches, sometimes the TypeScript cache can get mixed up and show some invalid errors. If you run into TypeScript issues (invalid errors, or if it's taking too long to build types), here a few things to try.
+
+1. Build TypeScript references with the clean command.
+
+```
+node scripts/build_ts_refs --clean
+```
+
+2. Restore your repository to a totally fresh state by running `git clean`
+
+```
+# dry-run the clean to see what will be deleted
+git clean -fdxn -e /config -e /.vscode
+
+# review the files which will be deleted, consider adding some more excludes (-e)
+# re-run without the dry-run (-n) flag to actually delete the files
+```

docs/CHANGELOG.asciidoc

@@ -231,6 +231,18 @@ The `logging.useUTC` setting has been removed. For more information, refer to {k
 The default timezone is UTC. To change the timezone, set `logging.timezone: false` in kibana.yml. Change the timezone when the system, such as a docker container, is configured for a nonlocal timezone.  
 ====  
 
+[discrete]
+[[breaking-32049]]
+.Removed environment variables `CONFIG_PATH` and `DATA_PATH`
+[%collapsible]
+====
+*Details* +
+The environment variables `CONFIG_PATH` and `DATA_PATH` have been removed.  For more information, refer to {kibana-pull}32049[#32049]
+
+*Impact* +
+Use the environment variable `KBN_PATH_CONF` instead of `CONFIG_PATH`.  Use the setting `path.data` instead of `DATA_PATH`.
+====
+
 // end::notable-breaking-changes[]
 
 [float]

docs/apm/correlations.asciidoc

@@ -58,68 +58,34 @@ out, you can begin viewing sample traces to continue your investigation.
 [[correlations-error-rate]]
 ==== Find failed transaction correlations
 
-The correlations on the *Error rate* tab help you discover which fields are
-contributing to failed transactions.
-
-By default, a number of attributes commonly known to cause performance issues,
-like version, infrastructure, and location, are included, but all are completely
-customizable to your APM data. Find something interesting? A quick click of a 
-button will auto-query your data as you work to resolve the underlying issue.
-
-The error rate over time chart visualizes the change in error rate over the selected time frame.
-Correlated attributes are sorted by _Impact_–a visual representation of the
-{ref}/search-aggregations-bucket-significantterms-aggregation.html[significant terms aggregation]
-score that powers correlations.
-Attributes with a high impact, or attributes present in a large percentage of failed transactions,
-may contribute to increased error rates.
-
-To find error rate correlations, hover over each potentially correlated attribute to
-compare the error rate distribution of transactions with and without the selected attribute.
-
-For example, in the screenshot below, the field `url.original` and value `http://localhost:3100...`
-existed in 100% of failed transactions between 6:00 and 10:30.
+beta::[]
+
+The correlations on the *Failed transaction correlations* tab help you discover
+which attributes are most influential in distinguishing between transaction
+failures and successes. In this context, the success or failure of a transaction
+is determined by its {ecs-ref}/ecs-event.html#field-event-outcome[event.outcome]
+value. For example, APM agents set the `event.outcome` to `failure` when an HTTP
+transaction returns a `5xx` status code.
+
+// The chart highlights the failed transactions in the overall latency distribution for the transaction group. 
+If there are attributes that have a statistically significant correlation with
+failed transactions, they are listed in a table. The table is sorted by scores,
+which are mapped to high, medium, or low impact levels. Attributes with high
+impact levels are more likely to contribute to failed transactions.
+//  By default, the attribute with the highest score is added to the chart. To see a different attribute in the chart, hover over its row in the table.
+
+For example, in the screenshot below, the field
+`kubernetes.pod.name` and value `frontend-node-59dff47885-fl5lb` has a medium
+impact level and existed in 19% of the failed transactions.
 
 [role="screenshot"]
-image::apm/images/error-rate-hover.png[Correlations errors hover effect]
-
-Select the `+` filter to create a new query in the {apm-app} for transactions with
-`url.original: http://localhost:3100...`. With the "noise" now filtered out,
-you can begin viewing sample traces to continue your investigation.
-
-As you sift through erroneous transactions, you'll likely notice other interesting attributes.
-Return to the correlations fly-out and select *Customize fields* to search on these new attributes.
-You may need to do this a few times–each time filtering out more and more noise and bringing you
-closer to a diagnosis.
-
-[discrete]
-[[correlations-customize-fields]]
-=====  Customize fields
-
-By default, a handful of attributes commonly known to cause performance issues
-are included in the analysis on the *Error rate* tab. You can add and remove
-fields under the **Customize fields** dropdown.
-
-The following fields are selected by default. To keep the default list
-manageable, only the first six matching fields with wildcards are used.
-
-**Frontend (RUM) agent:**
-
-* `labels.*`
-* `user.*`
-* `user_agent.name`
-* `user_agent.os.name`
-* `url.original`
-
-**Backend agents:**
+image::apm/images/correlations-failed-transactions.png[Failed transaction correlations]
 
-* `labels.*`
-* `host.ip`
-* `service.node.name`
-* `service.version`
+TIP: Some details, such as the failure and success percentages, are available
+only when the
+<<observability-enable-inspect-es-queries,observability:enableInspectEsQueries>>
+advanced setting is enabled.
 
-[TIP]
-====
-* Want to start over? Select **reset** to clear your customizations.
-* The *Latency* tab does not have a **Customize fields** dropdown, since it
-automatically considers all relevant fields in the transactions.
-====
+Select the `+` filter to create a new query in the {apm-app} for transactions
+with this attribute. You might do his for multiple attributes--each time
+filtering out more and more noise and bringing you closer to a diagnosis.

docs/development/core/public/kibana-plugin-core-public.doclinksstart.links.md

@@ -190,6 +190,9 @@ readonly links: {
         }>;
         readonly observability: Readonly<{
             guide: string;
+            infrastructureThreshold: string;
+            logsThreshold: string;
+            metricsThreshold: string;
             monitorStatus: string;
             monitorUptime: string;
             tlsCertificate: string;

docs/management/action-types.asciidoc

@@ -135,5 +135,14 @@ image::images/connectors-with-missing-secrets.png[Connectors with missing secret
 For out-of-the-box and standardized connectors, you can <<preconfigured-connector-example, preconfigure connectors>>
 before {kib} starts.
 
+[float]
+[[montoring-connectors]]
+=== Monitoring connectors
+
+The <<task-manager-health-monitoring,Task Manager health API>> helps you understand the performance of all tasks in your environment.
+However, if connectors fail to execute, they will report as successful to Task Manager. The failure stats will not
+accurately depict the performance of connectors.
+
+For more information on connector successes and failures, refer to the <<event-log-index,Event log index>>.
 
 include::connectors/index.asciidoc[]

docs/management/manage-index-patterns.asciidoc

@@ -7,60 +7,60 @@ To customize the data fields in your index pattern, you can add runtime fields t
 [[runtime-fields]]
 === Explore your data with runtime fields 
 
-Runtime fields are fields that you add to documents after you've ingested, and are evaluated at query time. With runtime fields, you allow for a smaller index and faster ingest time so that you can use less resources and reduce your operating costs. You can use runtime fields anywhere index patterns are used.
+Runtime fields are fields that you add to documents after you've ingested your data, and are evaluated at query time. With runtime fields, you allow for a smaller index and faster ingest time so that you can use less resources and reduce your operating costs. You can use runtime fields anywhere index patterns are used, for example, you can explore runtime fields in *Discover* and create visualizations with runtime fields for your dashboard.
 
-When you use runtime fields, you can:
+With runtime fields, you can:
 
-* Define fields for a specific use without modifying the underlying schema. 
+* Define fields for a specific use case without modifying the underlying schema. 
 
 * Override the returned values from index fields.
 
-* Start working on your data without first understanding the structure. 
+* Start working on your data without understanding the structure. 
 
 * Add fields to existing documents without reindexing your data. 
 
-* Explore runtime field data in *Discover*.
-
-* Create visualizations with runtime field data using *Lens*, *Maps*, and *TSVB*.
-
 WARNING: Runtime fields can impact {kib} performance. When you run a query, {es} uses the fields you index first to shorten the response time. 
 Index the fields that you commonly search for and filter on, such as `timestamp`, then use runtime fields to limit the number of fields {es} uses to calculate values. 
 
-For more information, refer to {ref}/runtime.html[Runtime fields].
+For detailed information on how to use runtime fields with {es}, refer to {ref}/runtime.html[Runtime fields].
 
 [float]
 [[create-runtime-fields]]
-==== Create runtime fields
+==== Add runtime fields
 
-Create runtime fields in your index patterns, or create runtime fields in *Discover* and *Lens*. 
+To add runtime fields to your index patterns, open the index pattern you want to change, then define the field values by emitting a single value using the {ref}/modules-scripting-painless.html[Painless scripting language]. You can also add runtime fields in <<add-field-in-discover,*Discover*>> and <<change-the-fields,*Lens*>>.
 
 . Open the main menu, then click *Stack Management > Index Patterns*.
 
 . Select the index pattern you want to add the runtime field to, then click *Add field*.
 
-. Enter a *Name* for the runtime field, then select the field *Type*.
+. Enter the field *Name*, then select the *Type*.
+
+. Select *Set custom label*, then enter the label you want to display where the index pattern is used, such as *Discover*.
+
+. Select *Set value*, then define the script. The script must match the *Type*, or the index pattern fails anywhere it is used.
+
+. To help you define the script, use the *Preview*:
+
+* To view the other available fields, use the *Document ID* arrows.
+
+* To filter the fields list, enter the keyword in *Filter fields*.
 
-. Select *Set value*, then define the field value by emitting a single value using the {ref}/modules-scripting-painless.html[Painless scripting language].
-+
-The script must match the field *Type*, or the script fails.
+* To pin frequently used fields to the top of the list, hover over the field, then click image:images/stackManagement-indexPatterns-pinRuntimeField-7.15.png[Icon to pin field to the top of the list].
 
 . Click *Create field*.
-+
-For information on how to create runtime fields in *Discover*, refer to <<add-field-in-discover,Add a field>>.
-+
-For information on how to create runtime fields in *Lens*, refer to <<change-the-fields,Change the fields list>>.
 
 [float]
 [[runtime-field-examples]]
 ==== Runtime field examples
 
-Try the runtime field examples on your own using the *Sample web logs* data index pattern. 
+Try the runtime field examples on your own using the <<gs-get-data-into-kibana,*Sample web logs*>> data index pattern. 
 
 [float]
 [[simple-hello-world-example]]
 ==== Return a keyword value
 
-To return `Hello World!` value:
+Return `Hello World!`:
 
 [source,text]
 ----
@@ -101,7 +101,7 @@ emit("");
 [[replace-nulls-with-blanks]]
 ===== Replace nulls with blanks
 
-Replace null values with none values:
+Replace `null` values with `None`:
 
 [source,text]
 ----
@@ -115,7 +115,7 @@ else {
 }
 ----
 
-Specify operating system condition:
+Specify the operating system condition:
 
 [source,text]
 ----

docs/user/production-considerations/task-manager-health-monitoring.asciidoc

@@ -111,6 +111,7 @@ a| Runtime
 
 | This section tracks excution performance of Task Manager, tracking task _drift_, worker _load_, and execution stats broken down by type, including duration and execution results.
 
+
 a| Capacity Estimation
 
 | This section provides a rough estimate about the sufficiency of its capacity. As the name suggests, these are estimates based on historical data and should not be used as predictions. Use these estimations when following the Task Manager <<task-manager-scaling-guidance>>.
@@ -123,6 +124,14 @@ The root `status` indicates the `status` of the system overall.
 
 The Runtime `status` indicates whether task executions have exceeded any of the <<task-manager-configuring-health-monitoring,configured health thresholds>>. An `OK` status means none of the threshold have been exceeded. A `Warning` status means that at least one warning threshold has been exceeded. An `Error` status means that at least one error threshold has been exceeded.
 
+[IMPORTANT]
+==============================================
+Some tasks (such as <<action-types,connectors>>) will incorrectly report their status as successful even if the task failed.
+The runtime and workload block will return data about success and failures and will not take this into consideration.
+
+To get a better sense of action failures, please refer to the <<event-log-index,Event log index>> for more accurate context into failures and successes.
+==============================================
+
 The Capacity Estimation `status` indicates the sufficiency of the observed capacity. An `OK` status means capacity is sufficient. A `Warning` status means that capacity is sufficient for the scheduled recurring tasks, but non-recurring tasks often cause the cluster to exceed capacity. An `Error` status means that there is insufficient capacity across all types of tasks.
 
 By monitoring the `status` of the system overall, and the `status` of specific task types of interest, you can evaluate the health of the {kib} Task Management system.