From df6c2cab88e67adf8c50debd39e22422464e6700 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Istv=C3=A1n=20Zolt=C3=A1n=20Szab=C3=B3?= Date: Wed, 24 Mar 2021 11:59:10 +0100 Subject: [PATCH] [DOCS] Reorganizes Transforms limitations (#70638) (#70807) --- docs/reference/transform/limitations.asciidoc | 174 +++++++++--------- 1 file changed, 92 insertions(+), 82 deletions(-) diff --git a/docs/reference/transform/limitations.asciidoc b/docs/reference/transform/limitations.asciidoc index 0f7bc02c40912..52504a57ef701 100644 --- a/docs/reference/transform/limitations.asciidoc +++ b/docs/reference/transform/limitations.asciidoc @@ -7,55 +7,88 @@ ++++ The following limitations and known problems apply to the {version} release of -the Elastic {transform} feature: +the Elastic {transform} feature. The limitations are grouped into the following +categories: + +* <> apply to the configuration process of the + {transforms}. +* <> affect the behavior of the {transforms} + that are running. +* <> only apply to {transforms} managed via the user + interface. + [discrete] -[[transform-space-limitations]] -== {transforms-cap} are visible in all {kib} spaces +[[transform-config-limitations]] +== Configuration limitations -{kibana-ref}/xpack-spaces.html[Spaces] enable you to organize your source and -destination indices and other saved objects in {kib} and to see only the objects -that belong to your space. However, this limited scope does not apply to -{transforms}; they are visible in all spaces. +[discrete] +[[transforms-ccs-limitation]] +=== {transforms-cap} support {ccs} if the remote cluster is configured properly + +If you use <>, the remote cluster must +support the search and aggregations you use in your {transforms}. +{transforms-cap} validate their configuration; if you use {ccs} and the +validation fails, make sure that the remote cluster supports the query and +aggregations you use. [discrete] -[[transform-ui-limitation]] -== {transforms-cap} UI will not work during a rolling upgrade from 7.2 +[[transform-painless-limitation]] +=== Using scripts in {transforms} -If your cluster contains mixed version nodes, for example during a rolling -upgrade from 7.2 to a newer version, and {transforms} have been created in 7.2, -the {transforms} UI (earler {dataframe} UI) will not work. Please wait until all -nodes have been upgraded to the newer version before using the {transforms} UI. +{transforms-cap} support scripting in every case when aggregations support them. +However, there are certain factors you might want to consider when using scripts +in {transforms}: +* {transforms-cap} cannot deduce index mappings for output fields when the + fields are created by a script. In this case, you might want to create the + mappings of the destination index yourself prior to creating the transform. + +* Scripted fields may increase the runtime of the {transform}. + +* {transforms-cap} cannot optimize queries when you use scripts for all the + groupings defined in `group_by`, you will receive a warning message when you + use scripts this way. + [discrete] -[[transform-rolling-upgrade-limitation]] -== {transforms-cap} reassignment suspended during a rolling upgrade from 7.2 and 7.3 +[[transform-runtime-field-limitation]] +=== {transforms-cap} perform better on indexed fields -If your cluster contains mixed version nodes, for example during a rolling -upgrade from 7.2 or 7.3 to a newer version, {transforms} whose nodes are stopped -will not be reassigned until the upgrade is complete. After the upgrade is done, -{transforms} resume automatically; no action is required. +{transforms-cap} sort data by a user-defined time field, which is frequently +accessed. If the time field is a {ref}/runtime.html[runtime field], the +performance impact of calculating field values at query time can significantly +slow the {transform}. Use an indexed field as a time field when using +{transforms}. [discrete] -[[transform-datatype-limitations]] -== {dataframe-cap} data type limitation +[[transform-scheduling-limitations]] +=== {ctransform-cap} scheduling limitations + +A {ctransform} periodically checks for changes to source data. The functionality +of the scheduler is currently limited to a basic periodic timer which can be +within the `frequency` range from 1s to 1h. The default is 1m. This is designed +to run little and often. When choosing a `frequency` for this timer consider +your ingest rate along with the impact that the {transform} +search/index operations has other users in your cluster. Also note that retries +occur at `frequency` interval. -{dataframes-cap} do not (yet) support fields containing arrays – in the UI or -the API. If you try to create one, the UI will fail to show the source index -table. [discrete] -[[transform-kibana-limitations]] -== Up to 1,000 {transforms} are supported +[[transform-operational-limitations]] +== Operational limitations + +[discrete] +[[transform-rolling-upgrade-limitation]] +=== {transforms-cap} reassignment suspended during a rolling upgrade from 7.2 and 7.3 -A single cluster will support up to 1,000 {transforms}. When using the -<> a total `count` of {transforms} -is returned. Use the `size` and `from` parameters to enumerate through the full -list. +If your cluster contains mixed version nodes, for example during a rolling +upgrade from 7.2 or 7.3 to a newer version, {transforms} whose nodes are stopped +will not be reassigned until the upgrade is complete. After the upgrade is done, +{transforms} resume automatically; no action is required. [discrete] [[transform-aggresponse-limitations]] -== Aggregation responses may be incompatible with destination index mappings +=== Aggregation responses may be incompatible with destination index mappings When a {transform} is first started, it will deduce the mappings required for the destination index. This process is based on the field types of @@ -80,7 +113,7 @@ derived from scripts that use dynamic mappings. [discrete] [[transform-batch-limitations]] -== Batch {transforms} may not account for changed documents +=== Batch {transforms} may not account for changed documents A batch {transform} uses a <> @@ -91,7 +124,7 @@ results may not include these changes. [discrete] [[transform-consistency-limitations]] -== {ctransform-cap} consistency does not account for deleted or updated documents +=== {ctransform-cap} consistency does not account for deleted or updated documents While the process for {transforms} allows the continual recalculation of the {transform} as new data is being ingested, it does also have some limitations. @@ -115,7 +148,7 @@ viewing the destination index. [discrete] [[transform-deletion-limitations]] -== Deleting a {transform} does not delete the destination index or {kib} index pattern +=== Deleting a {transform} does not delete the destination index or {kib} index pattern When deleting a {transform} using `DELETE _transform/index` neither the destination index nor the {kib} index pattern, should one have been @@ -123,7 +156,7 @@ created, are deleted. These objects must be deleted separately. [discrete] [[transform-aggregation-page-limitations]] -== Handling dynamic adjustment of aggregation page size +=== Handling dynamic adjustment of aggregation page size During the development of {transforms}, control was favoured over performance. In the design considerations, it is preferred for the {transform} to take longer @@ -152,7 +185,7 @@ its minimum, then the {transform} will be set to a failed state. [discrete] [[transform-dynamic-adjustments-limitations]] -== Handling dynamic adjustments for many terms +=== Handling dynamic adjustments for many terms For each checkpoint, entities are identified that have changed since the last time the check was performed. This list of changed entities is supplied as a @@ -172,21 +205,9 @@ is 65536. If `max_page_search_size` exceeds `index.max_terms_count` the Using smaller values for `max_page_search_size` may result in a longer duration for the {transform} checkpoint to complete. -[discrete] -[[transform-scheduling-limitations]] -== {ctransform-cap} scheduling limitations - -A {ctransform} periodically checks for changes to source data. The functionality -of the scheduler is currently limited to a basic periodic timer which can be -within the `frequency` range from 1s to 1h. The default is 1m. This is designed -to run little and often. When choosing a `frequency` for this timer consider -your ingest rate along with the impact that the {transform} -search/index operations has other users in your cluster. Also note that retries -occur at `frequency` interval. - [discrete] [[transform-failed-limitations]] -== Handling of failed {transforms} +=== Handling of failed {transforms} Failed {transforms} remain as a persistent task and should be handled appropriately, either by deleting it or by resolving the root cause of the @@ -197,7 +218,7 @@ When using the API to delete a failed {transform}, first stop it using [discrete] [[transform-availability-limitations]] -== {ctransforms-cap} may give incorrect results if documents are not yet available to search +=== {ctransforms-cap} may give incorrect results if documents are not yet available to search After a document is indexed, there is a very small delay until it is available to search. @@ -214,7 +235,7 @@ issue will occur. [discrete] [[transform-date-nanos]] -== Support for date nanoseconds data type +=== Support for date nanoseconds data type If your data uses the <>, aggregations are nonetheless on millisecond resolution. This limitation also affects the @@ -223,7 +244,7 @@ aggregations in your {transforms}. [discrete] [[transform-data-streams-destination]] -== Data streams as destination indices are not supported +=== Data streams as destination indices are not supported {transforms-cap} update data in the destination index which requires writing into the destination. <> are designed to be append-only, which @@ -234,7 +255,7 @@ this reason, data streams are not supported as destination indices for [discrete] [[transform-ilm-destination]] -== ILM as destination index may cause duplicated documents +=== ILM as destination index may cause duplicated documents <> is not recommended to use as a {transform} destination index. {transforms-cap} update documents in the current destination, @@ -248,40 +269,29 @@ documents if your {transform} contains a `group_by` based on `date_histogram`. [discrete] -[[transform-painless-limitation]] -== Using scripts in {transforms} +[[transform-ui-limitations]] +== Limitations in {kib} -{transforms-cap} support scripting in every case when aggregations support them. -However, there are certain factors you might want to consider when using scripts -in {transforms}: - -* {transforms-cap} cannot deduce index mappings for output fields when the - fields are created by a script. In this case, you might want to create the - mappings of the destination index yourself prior to creating the transform. +[discrete] +[[transform-space-limitations]] +=== {transforms-cap} are visible in all {kib} spaces -* Scripted fields may increase the runtime of the {transform}. - -* {transforms-cap} cannot optimize queries when you use scripts for all the - groupings defined in `group_by`, you will receive a warning message when you - use scripts this way. - +{kibana-ref}/xpack-spaces.html[Spaces] enable you to organize your source and +destination indices and other saved objects in {kib} and to see only the objects +that belong to your space. However, this limited scope does not apply to +{transforms}; they are visible in all spaces. [discrete] -[[transform-runtime-field-limitation]] -=== {transforms-cap} perform better on indexed fields - -{transforms-cap} sort data by a user-defined time field, which is frequently -accessed. If the time field is a {ref}/runtime.html[runtime field], the -performance impact of calculating field values at query time can significantly -slow the {transform}. Use an indexed field as a time field when using -{transforms}. +[[transform-rolling-upgrade-ui-limitation]] +=== {transforms-cap} UI will not work during a rolling upgrade from 7.2 +If your cluster contains mixed version nodes, for example during a rolling +upgrade from 7.2 to a newer version, and {transforms} have been created in 7.2, +the {transforms} UI (earler {dataframe} UI) will not work. Please wait until all +nodes have been upgraded to the newer version before using the {transforms} UI. [discrete] -[[transforms-ccs-limitation]] -=== {transforms-cap} support {ccs} if the remote cluster is configured properly +[[transform-kibana-limitations]] +=== Up to 1,000 {transforms} are listed in {kib} -If you use <>, the remote cluster must -support the search and aggregations you use in your {transforms}. -{transforms-cap} validate their configuration; if you use {ccs} and the validation fails, -make sure that the remote cluster supports the query and aggregations you use. +The {transforms} management page in {kib} lists up to 1000 {transforms}. \ No newline at end of file