diff --git a/docs/reference/data-rollup-transform.asciidoc b/docs/reference/data-rollup-transform.asciidoc index 5fe08d6f0d577..413b7d89d8251 100644 --- a/docs/reference/data-rollup-transform.asciidoc +++ b/docs/reference/data-rollup-transform.asciidoc @@ -9,8 +9,12 @@ * <> + include::rollup/index.asciidoc[tag=rollup-intro] -* {stack-ov}/ml-dataframes.html[Transforming your data] +* <> ++ +include::transform/index.asciidoc[tag=transform-intro] -- include::rollup/index.asciidoc[] + +include::transform/index.asciidoc[] diff --git a/docs/reference/transform/api-quickref.asciidoc b/docs/reference/transform/api-quickref.asciidoc index 9d2590a1540ef..d2dff5c3021d3 100644 --- a/docs/reference/transform/api-quickref.asciidoc +++ b/docs/reference/transform/api-quickref.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] -[[df-api-quickref]] -== API quick reference +[[transform-api-quickref]] +=== API quick reference All {transform} endpoints have the following base: diff --git a/docs/reference/transform/apis/index.asciidoc b/docs/reference/transform/apis/index.asciidoc index e496401d340dd..80384af95980e 100644 --- a/docs/reference/transform/apis/index.asciidoc +++ b/docs/reference/transform/apis/index.asciidoc @@ -3,8 +3,6 @@ [[transform-apis]] == {transform-cap} APIs -See also {stack-ov}/ml-dataframes.html[{transforms-cap}]. - * <> * <> * <> diff --git a/docs/reference/transform/apis/put-transform.asciidoc b/docs/reference/transform/apis/put-transform.asciidoc index 49c3d7981ddb0..d72f2722f4e09 100644 --- a/docs/reference/transform/apis/put-transform.asciidoc +++ b/docs/reference/transform/apis/put-transform.asciidoc @@ -37,8 +37,7 @@ entities are defined by the set of `group_by` fields in the `pivot` object. You can also think of the destination index as a two-dimensional tabular data structure (known as a {dataframe}). The ID for each document in the {dataframe} is generated from a hash of the entity, so there is a unique row -per entity. For more information, see -{stack-ov}/ml-dataframes.html[{transforms-cap}]. +per entity. For more information, see <>. When the {transform} is created, a series of validations occur to ensure its success. For example, there is a check for the existence of the diff --git a/docs/reference/transform/checkpoints.asciidoc b/docs/reference/transform/checkpoints.asciidoc index 4c41b876b23c6..379834e8553d5 100644 --- a/docs/reference/transform/checkpoints.asciidoc +++ b/docs/reference/transform/checkpoints.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] -[[ml-transform-checkpoints]] -== How {transform} checkpoints work +[[transform-checkpoints]] +=== How {transform} checkpoints work ++++ How checkpoints work ++++ diff --git a/docs/reference/transform/ecommerce-example.asciidoc b/docs/reference/transform/ecommerce-tutorial.asciidoc similarity index 96% rename from docs/reference/transform/ecommerce-example.asciidoc rename to docs/reference/transform/ecommerce-tutorial.asciidoc index 7f8267baa1656..026127f97bac5 100644 --- a/docs/reference/transform/ecommerce-example.asciidoc +++ b/docs/reference/transform/ecommerce-tutorial.asciidoc @@ -1,11 +1,11 @@ [role="xpack"] [testenv="basic"] -[[ecommerce-dataframes]] -=== Transforming the eCommerce sample data +[[ecommerce-transforms]] +=== Tutorial: Transforming the eCommerce sample data beta[] -<> enable you to retrieve information +<> enable you to retrieve information from an {es} index, transform it, and store it in another index. Let's use the {kibana-ref}/add-sample-data.html[{kib} sample data] to demonstrate how you can pivot and summarize your data with {transforms}. @@ -23,7 +23,9 @@ You also need `read` and `view_index_metadata` index privileges on the source index and `read`, `create_index`, and `index` privileges on the destination index. -For more information, see <> and <>. +For more information, see +{stack-ov}/security-privileges.html[Security privileges] and +{stack-ov}/built-in-roles.html[Built-in roles]. -- . Choose your _source index_. diff --git a/docs/reference/transform/dataframe-examples.asciidoc b/docs/reference/transform/examples.asciidoc similarity index 97% rename from docs/reference/transform/dataframe-examples.asciidoc rename to docs/reference/transform/examples.asciidoc index 6c03ad3ecb3e9..6b15b0517e765 100644 --- a/docs/reference/transform/dataframe-examples.asciidoc +++ b/docs/reference/transform/examples.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] -[[dataframe-examples]] -== {transform-cap} examples +[[transform-examples]] +=== {transform-cap} examples ++++ Examples ++++ @@ -12,17 +12,14 @@ These examples demonstrate how to use {transforms} to derive useful insights from your data. All the examples use one of the {kibana-ref}/add-sample-data.html[{kib} sample datasets]. For a more detailed, step-by-step example, see -<>. +<>. -* <> * <> * <> * <> -include::ecommerce-example.asciidoc[] - [[example-best-customers]] -=== Finding your best customers +==== Finding your best customers In this example, we use the eCommerce orders sample dataset to find the customers who spent the most in our hypothetical webshop. Let's transform the data such @@ -106,7 +103,7 @@ navigate data from a customer centric perspective. In some cases, it can even make creating visualizations much simpler. [[example-airline]] -=== Finding air carriers with the most delays +==== Finding air carriers with the most delays In this example, we use the Flights sample dataset to find out which air carrier had the most delays. First, we filter the source data such that it excludes all @@ -193,7 +190,7 @@ or flight stats for any of the featured destination or origin airports. [[example-clientips]] -=== Finding suspicious client IPs by using scripted metrics +==== Finding suspicious client IPs by using scripted metrics With {transforms}, you can use {ref}/search-aggregations-metrics-scripted-metric-aggregation.html[scripted diff --git a/docs/reference/transform/index.asciidoc b/docs/reference/transform/index.asciidoc index 41ffd97ee3996..25a7f3dd407d8 100644 --- a/docs/reference/transform/index.asciidoc +++ b/docs/reference/transform/index.asciidoc @@ -1,27 +1,27 @@ [role="xpack"] -[[ml-dataframes]] -= Transforming data - -[partintro] --- +[[transforms]] +== Transforming data +// tag::transform-intro[] {transforms-cap} enable you to convert existing {es} indices into summarized -indices, which provide opportunities for new insights and analytics. For example, -you can use {transforms} to pivot your data into entity-centric indices that -summarize the behavior of users or sessions or other entities in your data. +indices, which provide opportunities for new insights and analytics. +// end::transform-intro[] +For example, you can use {transforms} to pivot your data into entity-centric +indices that summarize the behavior of users or sessions or other entities in +your data. -* <> -* <> -* <> -* <> -* <> -* <> --- +* <> +* <> +* <> +* <> +* <> +* <> include::overview.asciidoc[] include::usage.asciidoc[] include::checkpoints.asciidoc[] include::api-quickref.asciidoc[] -include::dataframe-examples.asciidoc[] +include::ecommerce-tutorial.asciidoc[] +include::examples.asciidoc[] include::troubleshooting.asciidoc[] include::limitations.asciidoc[] \ No newline at end of file diff --git a/docs/reference/transform/limitations.asciidoc b/docs/reference/transform/limitations.asciidoc index a97737464b3fc..70a3ffb25432b 100644 --- a/docs/reference/transform/limitations.asciidoc +++ b/docs/reference/transform/limitations.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] -[[dataframe-limitations]] -== {transform-cap} limitations +[[transform-limitations]] +=== {transform-cap} limitations [subs="attributes"] ++++ Limitations @@ -12,8 +12,8 @@ The following limitations and known problems apply to the 7.4 release of the Elastic {dataframe} feature: [float] -[[df-compatibility-limitations]] -=== Beta {transforms} do not have guaranteed backwards or forwards compatibility +[[transform-compatibility-limitations]] +==== Beta {transforms} do not have guaranteed backwards or forwards compatibility Whilst {transforms} are beta, it is not guaranteed that a {transform} created in a previous version of the {stack} will be able @@ -25,8 +25,8 @@ destination index. This is a normal {es} index and is not affected by the beta status. [float] -[[df-ui-limitation]] -=== {dataframe-cap} UI will not work during a rolling upgrade from 7.2 +[[transform-ui-limitation]] +==== {dataframe-cap} UI will not work during a rolling upgrade from 7.2 If your cluster contains mixed version nodes, for example during a rolling upgrade from 7.2 to a newer version, and {transforms} have been @@ -35,22 +35,22 @@ have been upgraded to the newer version before using the {dataframe} UI. [float] -[[df-datatype-limitations]] -=== {dataframe-cap} data type limitation +[[transform-datatype-limitations]] +==== {dataframe-cap} data type limitation {dataframes-cap} do not (yet) support fields containing arrays – in the UI or the API. If you try to create one, the UI will fail to show the source index table. [float] -[[df-ccs-limitations]] -=== {ccs-cap} is not supported +[[transform-ccs-limitations]] +==== {ccs-cap} is not supported {ccs-cap} is not supported for {transforms}. [float] -[[df-kibana-limitations]] -=== Up to 1,000 {transforms} are supported +[[transform-kibana-limitations]] +==== Up to 1,000 {transforms} are supported A single cluster will support up to 1,000 {transforms}. When using the @@ -59,8 +59,8 @@ When using the enumerate through the full list. [float] -[[df-aggresponse-limitations]] -=== Aggregation responses may be incompatible with destination index mappings +[[transform-aggresponse-limitations]] +==== Aggregation responses may be incompatible with destination index mappings When a {transform} is first started, it will deduce the mappings required for the destination index. This process is based on the field types of @@ -77,8 +77,8 @@ workaround, you may define custom mappings prior to starting the {ref}/indices-templates.html[define an index template]. [float] -[[df-batch-limitations]] -=== Batch {transforms} may not account for changed documents +[[transform-batch-limitations]] +==== Batch {transforms} may not account for changed documents A batch {transform} uses a {ref}/search-aggregations-bucket-composite-aggregation.html[composite aggregation] @@ -88,8 +88,8 @@ do not yet support a search context, therefore if the source data is changed results may not include these changes. [float] -[[df-consistency-limitations]] -=== {cdataframe-cap} consistency does not account for deleted or updated documents +[[transform-consistency-limitations]] +==== {cdataframe-cap} consistency does not account for deleted or updated documents While the process for {transforms} allows the continual recalculation of the {transform} as new data is being ingested, it does also have @@ -114,16 +114,16 @@ updated when viewing the {dataframe} destination index. [float] -[[df-deletion-limitations]] -=== Deleting a {transform} does not delete the {dataframe} destination index or {kib} index pattern +[[transform-deletion-limitations]] +==== Deleting a {transform} does not delete the {dataframe} destination index or {kib} index pattern When deleting a {transform} using `DELETE _data_frame/transforms/index` neither the {dataframe} destination index nor the {kib} index pattern, should one have been created, are deleted. These objects must be deleted separately. [float] -[[df-aggregation-page-limitations]] -=== Handling dynamic adjustment of aggregation page size +[[transform-aggregation-page-limitations]] +==== Handling dynamic adjustment of aggregation page size During the development of {transforms}, control was favoured over performance. In the design considerations, it is preferred for the @@ -153,8 +153,8 @@ requested has been reduced to its minimum, then the {transform} will be set to a failed state. [float] -[[df-dynamic-adjustments-limitations]] -=== Handling dynamic adjustments for many terms +[[transform-dynamic-adjustments-limitations]] +==== Handling dynamic adjustments for many terms For each checkpoint, entities are identified that have changed since the last time the check was performed. This list of changed entities is supplied as a @@ -176,8 +176,8 @@ Using smaller values for `max_page_search_size` may result in a longer duration for the {transform} checkpoint to complete. [float] -[[df-scheduling-limitations]] -=== {cdataframe-cap} scheduling limitations +[[transform-scheduling-limitations]] +==== {cdataframe-cap} scheduling limitations A {cdataframe} periodically checks for changes to source data. The functionality of the scheduler is currently limited to a basic periodic timer which can be @@ -188,8 +188,8 @@ search/index operations has other users in your cluster. Also note that retries occur at `frequency` interval. [float] -[[df-failed-limitations]] -=== Handling of failed {transforms} +[[transform-failed-limitations]] +==== Handling of failed {transforms} Failed {transforms} remain as a persistent task and should be handled appropriately, either by deleting it or by resolving the root cause of the @@ -199,8 +199,8 @@ When using the API to delete a failed {transform}, first stop it using `_stop?force=true`, then delete it. [float] -[[df-availability-limitations]] -=== {cdataframes-cap} may give incorrect results if documents are not yet available to search +[[transform-availability-limitations]] +==== {cdataframes-cap} may give incorrect results if documents are not yet available to search After a document is indexed, there is a very small delay until it is available to search. diff --git a/docs/reference/transform/overview.asciidoc b/docs/reference/transform/overview.asciidoc index fa161f2e9ea3b..e3c852d8be941 100644 --- a/docs/reference/transform/overview.asciidoc +++ b/docs/reference/transform/overview.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] -[[ml-transform-overview]] -== {transform-cap} overview +[[transform-overview]] +=== {transform-cap} overview ++++ Overview ++++ diff --git a/docs/reference/transform/troubleshooting.asciidoc b/docs/reference/transform/troubleshooting.asciidoc index 9d76e93415db6..b453529a65fe2 100644 --- a/docs/reference/transform/troubleshooting.asciidoc +++ b/docs/reference/transform/troubleshooting.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] -[[dataframe-troubleshooting]] -== Troubleshooting {transforms} +[[transform-troubleshooting]] +=== Troubleshooting {transforms} [subs="attributes"] ++++ Troubleshooting @@ -9,7 +9,11 @@ Use the information in this section to troubleshoot common problems. -include::{stack-repo-dir}/help.asciidoc[tag=get-help] +For issues that you cannot fix yourself … we’re here to help. +If you are an existing Elastic customer with a support contract, please create +a ticket in the +https://support.elastic.co/customers/s/login/[Elastic Support portal]. +Or post in the https://discuss.elastic.co/[Elastic forum]. If you encounter problems with your {transforms}, you can gather more information from the following files and APIs: diff --git a/docs/reference/transform/usage.asciidoc b/docs/reference/transform/usage.asciidoc index 70dfe0f80b396..f78a0388bcb82 100644 --- a/docs/reference/transform/usage.asciidoc +++ b/docs/reference/transform/usage.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] -[[ml-transforms-usage]] -== When to use {transforms} +[[transform-usage]] +=== When to use {transforms} {es} aggregations are a powerful and flexible feature that enable you to summarize and retrieve complex insights about your data. You can summarize