From 2877a8dcbd1c336e3f43a1598fb2fcb306d58097 Mon Sep 17 00:00:00 2001 From: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com> Date: Fri, 3 Jun 2022 17:15:35 +1000 Subject: [PATCH 1/6] Add CCR limitation closes https://github.com/elastic/elasticsearch/issues/86121 --- docs/reference/ccr/limitations.asciidoc | 76 +++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 docs/reference/ccr/limitations.asciidoc diff --git a/docs/reference/ccr/limitations.asciidoc b/docs/reference/ccr/limitations.asciidoc new file mode 100644 index 0000000000000..cce7adba4e064 --- /dev/null +++ b/docs/reference/ccr/limitations.asciidoc @@ -0,0 +1,76 @@ +[role="xpack"] +[[ccr-limitations]] +== CCR Limitations + +As {ccr} is designed to replicate user-generated indices only, all +<>, <>, +<>, <> and +<> policies, <>, <> and +<> currently do not support getting +replicated via {ccr}. + +[discrete] + + +[[ccr-system-indices]] +=== System indices + +All <> are not designed to be replicated using +{ccr}. This includes: +[unordered] +APM:: All APM configurations stored in `.apm*`. +Elastic Security:: All SIEM objects. +EnterpriseSearch:: All enterprise Search system objects stored in `.ent-search*`. +Fleet:: All fleet agent actions and configurations stored in `.fleet*`. +Kibana:: Saved objects (including visualization, reporting, alerts etc) +currently stored in `.kibana*` or `.reporting*`. +Machine Learning:: All machine learning jobs. +Watcher:: All watcher definition and execution history. + +[[ccr-index-templates]] +=== Index templates + +{ccr} does not support replicating index templates across clusters, please +replicate them to a remote cluster manually. + +[[ccr-ilm-slm-policies]] +=== ILM/SLM Policies + +Both <> and <> polices are stored locally in cluster state. {ccr} does not +support replicating these policies, please replicate them to a remote cluster +manually. + +[[ccr-snapshot-settings]] +=== Snapshot Settings + +Snapshot repository settings are stored locally in the cluster state. {ccr} +does not support replicating these settings, please replicate them to a remote +cluster manually. + +[[ccr-searchable-snapshots]] +=== Searchable Snapshots + +The actual data for <> indices are +stored in the snapshot repository while indices are either partially-cached or +fully-cached on the Elasticearch nodes, {ccr} will not replicate the searchable +snapshot correctly. You can still set up snapshot repositories on the remote +cluster and use the same ilm policy to move data into cold or frozen tier thus +achieving searchable snapshots in the remote cluster. Data stored in the cold +and frozen tier are expected to be read-only, thus only replicate the data in +the hot and warm tier and moving them into searchable snapshots in the cold or +frozen tier will achieve the same effect. See <> for more details. + +[[ccr-user-role-permission-mapping]] +=== User/Role Permission and Mapping + +User permissions and role mappings are stored as part of the cluster state. +{ccr} does not support replicating these settings, please replicate them to a +remote cluster manually. + +[[ccr-cluster-settings]] +=== Cluster Settings +{ccr} does not support replicating cluster settings, please replicate them to a +remote cluster manually. From f6196cf1cf5616242a8e784d6567bcdd978c25f4 Mon Sep 17 00:00:00 2001 From: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com> Date: Tue, 14 Jun 2022 23:26:11 +1000 Subject: [PATCH 2/6] Add restored index auto follow pattern restriction https://github.com/elastic/elasticsearch/issues/87055 --- docs/reference/ccr/limitations.asciidoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/reference/ccr/limitations.asciidoc b/docs/reference/ccr/limitations.asciidoc index cce7adba4e064..81c36ba565e33 100644 --- a/docs/reference/ccr/limitations.asciidoc +++ b/docs/reference/ccr/limitations.asciidoc @@ -74,3 +74,6 @@ remote cluster manually. === Cluster Settings {ccr} does not support replicating cluster settings, please replicate them to a remote cluster manually. + +=== Restored index +{ccr} auto follow pattern will not automatically replicated restored indices From aa7409e77e5eda3d9ebbd2dd8bee33f0449217ce Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Thu, 7 Jul 2022 13:20:53 -0400 Subject: [PATCH 3/6] Moving content to existing CCR page + several changes --- docs/reference/ccr/index.asciidoc | 89 +++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/docs/reference/ccr/index.asciidoc b/docs/reference/ccr/index.asciidoc index f42f0d0f51bf0..b2442f28c6694 100644 --- a/docs/reference/ccr/index.asciidoc +++ b/docs/reference/ccr/index.asciidoc @@ -307,6 +307,95 @@ and use {ccr}: * <> * <> +[discrete] +[[ccr-limitations]] +=== {ccr-cap} limitations +{ccr-cap} is designed to replicate user-generated indices only and doesn't +currently replicate any of the following: + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +[discrete] +[[ccr-system-indices]] +==== System indices +<> store configurations and internal data for +Elastic features. Generally, system indices are reserved only for internal use +by these features. {ccr-cap} doesn't replicate these indices as a security +precaution. + +[discrete] +[[ccr-ml-jobs]] +==== Machine learning jobs +{ccr-cap} doesn't support replicating +{ml-docs}/machine-learning-intro.html[machine learning jobs] across clusters. If +you want to replicate index templates, replicate them to a remote cluster manually. + +[discrete] +[[ccr-index-templates]] +==== Index templates +{ccr-cap} doesn't support replicating <> across +clusters. If you want to replicate index templates, replicate them to a remote +cluster manually. + +[discrete] +[[ccr-ilm-slm-policies]] +==== ILM/SLM policies +Both <> and +<> polices are stored locally in the +cluster state. If you want to replicate these policies, you must replicate them +to a remote cluster manually. + +[discrete] +[[ccr-user-role-permission-mapping]] +==== User permissions and role mappings +{ref}/mapping-roles.html[User permissions and role mappings] are stored as part +of the cluster state. If you want to replicate these settings, you must +replicate them to a remote cluster manually. + +[discrete] +[[ccr-snapshot-settings]] +==== Snapshot repository settings +<> are stored +locally in the cluster state. If you want to replicate these settings, you must +{ccr} replicate them to a remote cluster manually. + +[discrete] +[[ccr-cluster-settings]] +==== Cluster settings +If you want to replicate <>, you must replicate +them to a remote cluster manually. + +[discrete] +[[ccr-searchable-snapshots]] +==== Searchable snapshots +The actual data for <> indices are +stored in the snapshot repository. While indices are either partially-cached or +fully-cached on the {es} nodes, {ccr} will not replicate the searchable +snapshot correctly. You can still set up snapshot repositories on the remote +cluster and use the same {ilm} policy to move data into the cold or frozen tiers +to achieve searchable snapshots in the remote cluster. + +Data stored in the cold and frozen tier is expected to be read-only. To achieve +searchable snapshots in these tiers, replicate the data in the hot and warm +tiers and then move them into searchable snapshots in the cold or frozen tiers. +Use the <> to take a +snapshot of the indices and mount them as a searchable snapshot. + +[discrete] +[[ccr-restored-indices]] +==== Restored indices +{ccr-cap} auto-follow patterns don't automatically replicate restored indices. If +you want to replicate restored indices, you must replicate them to a remote +cluster manually. + include::getting-started.asciidoc[] include::managing.asciidoc[] include::auto-follow.asciidoc[] From aa1641fb26f43ad3452314096445c7babb2b33d1 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Tue, 4 Oct 2022 11:27:09 -0400 Subject: [PATCH 4/6] Remove sections to consolidate limitation information --- docs/reference/ccr/index.asciidoc | 104 ++++++------------------------ 1 file changed, 21 insertions(+), 83 deletions(-) diff --git a/docs/reference/ccr/index.asciidoc b/docs/reference/ccr/index.asciidoc index b2442f28c6694..634cfe4ec9bfb 100644 --- a/docs/reference/ccr/index.asciidoc +++ b/docs/reference/ccr/index.asciidoc @@ -310,91 +310,29 @@ and use {ccr}: [discrete] [[ccr-limitations]] === {ccr-cap} limitations -{ccr-cap} is designed to replicate user-generated indices only and doesn't +{ccr-cap} is designed to replicate user-generated indices only, and doesn't currently replicate any of the following: -* <> -* <> -* <> -* <> -* <> -* <> -* <> -* <> -* <> - -[discrete] -[[ccr-system-indices]] -==== System indices -<> store configurations and internal data for -Elastic features. Generally, system indices are reserved only for internal use -by these features. {ccr-cap} doesn't replicate these indices as a security -precaution. - -[discrete] -[[ccr-ml-jobs]] -==== Machine learning jobs -{ccr-cap} doesn't support replicating -{ml-docs}/machine-learning-intro.html[machine learning jobs] across clusters. If -you want to replicate index templates, replicate them to a remote cluster manually. - -[discrete] -[[ccr-index-templates]] -==== Index templates -{ccr-cap} doesn't support replicating <> across -clusters. If you want to replicate index templates, replicate them to a remote -cluster manually. - -[discrete] -[[ccr-ilm-slm-policies]] -==== ILM/SLM policies -Both <> and -<> polices are stored locally in the -cluster state. If you want to replicate these policies, you must replicate them -to a remote cluster manually. - -[discrete] -[[ccr-user-role-permission-mapping]] -==== User permissions and role mappings -{ref}/mapping-roles.html[User permissions and role mappings] are stored as part -of the cluster state. If you want to replicate these settings, you must -replicate them to a remote cluster manually. - -[discrete] -[[ccr-snapshot-settings]] -==== Snapshot repository settings -<> are stored -locally in the cluster state. If you want to replicate these settings, you must -{ccr} replicate them to a remote cluster manually. - -[discrete] -[[ccr-cluster-settings]] -==== Cluster settings -If you want to replicate <>, you must replicate -them to a remote cluster manually. - -[discrete] -[[ccr-searchable-snapshots]] -==== Searchable snapshots -The actual data for <> indices are -stored in the snapshot repository. While indices are either partially-cached or -fully-cached on the {es} nodes, {ccr} will not replicate the searchable -snapshot correctly. You can still set up snapshot repositories on the remote -cluster and use the same {ilm} policy to move data into the cold or frozen tiers -to achieve searchable snapshots in the remote cluster. - -Data stored in the cold and frozen tier is expected to be read-only. To achieve -searchable snapshots in these tiers, replicate the data in the hot and warm -tiers and then move them into searchable snapshots in the cold or frozen tiers. -Use the <> to take a -snapshot of the indices and mount them as a searchable snapshot. - -[discrete] -[[ccr-restored-indices]] -==== Restored indices -{ccr-cap} auto-follow patterns don't automatically replicate restored indices. If -you want to replicate restored indices, you must replicate them to a remote -cluster manually. +* <> +* {ml-docs}/machine-learning-intro.html[Machine learning jobs] +* <> +* <> and +<> polices +* {ref}/mapping-roles.html[User permissions and role mappings] +* <> +* <> +* <> +* <> + +If you want to replicate any of this data, you must replicate it to a remote +cluster manually. + +NOTE: Data for <> indices is stored in +the snapshot repository. {ccr-cap} won't replicate these indices completely, even +though they're either partially or fully-cached on the {es} nodes. To achieve +searchable snapshots in a remote cluster, configure snapshot repositories on +the remote cluster and use the same {ilm} policy from the local cluster to move +data into the cold or frozen tiers on the remote cluster. include::getting-started.asciidoc[] include::managing.asciidoc[] From f8bcbeb59b3fe56df0e33289b4f0e86fc9684c65 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Tue, 4 Oct 2022 11:35:55 -0400 Subject: [PATCH 5/6] Delete separate file --- docs/reference/ccr/limitations.asciidoc | 79 ------------------------- 1 file changed, 79 deletions(-) delete mode 100644 docs/reference/ccr/limitations.asciidoc diff --git a/docs/reference/ccr/limitations.asciidoc b/docs/reference/ccr/limitations.asciidoc deleted file mode 100644 index 81c36ba565e33..0000000000000 --- a/docs/reference/ccr/limitations.asciidoc +++ /dev/null @@ -1,79 +0,0 @@ -[role="xpack"] -[[ccr-limitations]] -== CCR Limitations - -As {ccr} is designed to replicate user-generated indices only, all -<>, <>, -<>, <> and -<> policies, <>, <> and -<> currently do not support getting -replicated via {ccr}. - -[discrete] - - -[[ccr-system-indices]] -=== System indices - -All <> are not designed to be replicated using -{ccr}. This includes: -[unordered] -APM:: All APM configurations stored in `.apm*`. -Elastic Security:: All SIEM objects. -EnterpriseSearch:: All enterprise Search system objects stored in `.ent-search*`. -Fleet:: All fleet agent actions and configurations stored in `.fleet*`. -Kibana:: Saved objects (including visualization, reporting, alerts etc) -currently stored in `.kibana*` or `.reporting*`. -Machine Learning:: All machine learning jobs. -Watcher:: All watcher definition and execution history. - -[[ccr-index-templates]] -=== Index templates - -{ccr} does not support replicating index templates across clusters, please -replicate them to a remote cluster manually. - -[[ccr-ilm-slm-policies]] -=== ILM/SLM Policies - -Both <> and <> polices are stored locally in cluster state. {ccr} does not -support replicating these policies, please replicate them to a remote cluster -manually. - -[[ccr-snapshot-settings]] -=== Snapshot Settings - -Snapshot repository settings are stored locally in the cluster state. {ccr} -does not support replicating these settings, please replicate them to a remote -cluster manually. - -[[ccr-searchable-snapshots]] -=== Searchable Snapshots - -The actual data for <> indices are -stored in the snapshot repository while indices are either partially-cached or -fully-cached on the Elasticearch nodes, {ccr} will not replicate the searchable -snapshot correctly. You can still set up snapshot repositories on the remote -cluster and use the same ilm policy to move data into cold or frozen tier thus -achieving searchable snapshots in the remote cluster. Data stored in the cold -and frozen tier are expected to be read-only, thus only replicate the data in -the hot and warm tier and moving them into searchable snapshots in the cold or -frozen tier will achieve the same effect. See <> for more details. - -[[ccr-user-role-permission-mapping]] -=== User/Role Permission and Mapping - -User permissions and role mappings are stored as part of the cluster state. -{ccr} does not support replicating these settings, please replicate them to a -remote cluster manually. - -[[ccr-cluster-settings]] -=== Cluster Settings -{ccr} does not support replicating cluster settings, please replicate them to a -remote cluster manually. - -=== Restored index -{ccr} auto follow pattern will not automatically replicated restored indices From dba14804a4eb83a1958fe6f6a306649cef6c6cc0 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Mon, 17 Oct 2022 15:17:40 -0400 Subject: [PATCH 6/6] Remove restored indices from list of things that aren't replicated --- docs/reference/ccr/index.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/ccr/index.asciidoc b/docs/reference/ccr/index.asciidoc index 634cfe4ec9bfb..f3e0640f3bf20 100644 --- a/docs/reference/ccr/index.asciidoc +++ b/docs/reference/ccr/index.asciidoc @@ -322,7 +322,6 @@ currently replicate any of the following: * <> * <> * <> -* <> If you want to replicate any of this data, you must replicate it to a remote cluster manually.