Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Add CCR limitation #87348

Merged
merged 7 commits into from
Oct 17, 2022
Merged

[DOCS] Add CCR limitation #87348

merged 7 commits into from
Oct 17, 2022

Conversation

Leaf-Lin
Copy link
Contributor

@Leaf-Lin Leaf-Lin commented Jun 3, 2022

Outlining CCR limitations in today's implementation.

closes #86121

The searchable snapshot part should also link to #67668 and #81238

Preview link: https://elasticsearch_87348.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/xpack-ccr.html#ccr-limitations

@Leaf-Lin Leaf-Lin added >docs General docs changes :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features Team:Docs Meta label for docs team labels Jun 3, 2022
@Leaf-Lin Leaf-Lin requested a review from lockewritesdocs June 3, 2022 07:18
@Leaf-Lin Leaf-Lin added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Jun 7, 2022
@lockewritesdocs
Copy link
Contributor

@Leaf-Lin, I moved this content to a section within the overall CCR context, moved the links to other pages into the separate sections, and made some additional edits. Let me know if these changes work for you.

@lockewritesdocs lockewritesdocs marked this pull request as ready for review July 7, 2022 17:22
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-docs (Team:Docs)

@lockewritesdocs
Copy link
Contributor

@elasticmachine run elasticsearch-ci/docs

@Leaf-Lin Leaf-Lin requested a review from henningandersen July 18, 2022 06:38
@elasticsearchmachine elasticsearchmachine changed the base branch from master to main July 22, 2022 23:06
@mark-vieira mark-vieira added v8.5.0 and removed v8.4.0 labels Jul 27, 2022
@bahaaldine
Copy link

@Leaf-Lin should we also add that CCR doesn't work across clusters that are individually configured with private link?

@lockewritesdocs lockewritesdocs removed the request for review from henningandersen August 3, 2022 18:44
@lockewritesdocs
Copy link
Contributor

Perhaps someone from @elastic/es-distributed can review these changes for accuracy.

@lockewritesdocs
Copy link
Contributor

@elasticmachine update branch

@kingherc kingherc added v8.4.3 and removed v8.4.2 labels Sep 15, 2022
@lockewritesdocs lockewritesdocs self-assigned this Oct 4, 2022
@lockewritesdocs
Copy link
Contributor

@Leaf-Lin, I consolidated this information considerably. I don't think that we need to get into the implementation details of where things are stored, and can instead list the data that isn't replicated with CCR and inform users that they'll need to manually replicate this information. Let me know if this change works for you, and if so, I'll merge.

Copy link
Contributor

@henningandersen henningandersen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one question, otherwise looking good.

* <<snapshots-register-repository,Snapshot repository settings>>
* <<modules-cluster,Cluster settings>>
* <<searchable-snapshots,Searchable snapshot>>
* <<restore-index-data-stream,Restored indices>>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this true, would auto-follow patterns not hit that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was referring to the problem described in this issue: #87055

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the conversation in #87055, it sounds like restored indices aren't replicated if the leader index is deleted and then restored, but the original replicated index still exists in the remote cluster. The restored index in the leader cluster gets a new UUID, so the auto-follow pattern on the remote cluster sees it as new index, but it's actually just a replicated index.

This sounds like an edge case that we could call attention to, but it seems like restored indices would be replicated if the underlying leader index is still intact. Is that true @henningandersen?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree @lockewritesdocs, it will follow just fine if the restore results in an unoccupied name on the follower. This is similar to any sort of naming conflict on the follower though a delete and restore on the leader side is possibly a common cause of this.

I'd opt to remove this, since it is not true that restored indices are not followed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Sounds good @henningandersen -- I've removed "Restored indices" from the list of things that aren't followed.

Copy link
Contributor

@henningandersen henningandersen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.3
8.5
8.4

Leaf-Lin added a commit to Leaf-Lin/elasticsearch that referenced this pull request Oct 17, 2022
* Add CCR limitation

closes elastic#86121

* Add restored index auto follow pattern restriction

elastic#87055

* Moving content to existing CCR page + several changes

* Remove sections to consolidate limitation information

* Delete separate file

* Remove restored indices from list of things that aren't replicated

Co-authored-by: Adam Locke <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
Leaf-Lin added a commit to Leaf-Lin/elasticsearch that referenced this pull request Oct 17, 2022
* Add CCR limitation

closes elastic#86121

* Add restored index auto follow pattern restriction

elastic#87055

* Moving content to existing CCR page + several changes

* Remove sections to consolidate limitation information

* Delete separate file

* Remove restored indices from list of things that aren't replicated

Co-authored-by: Adam Locke <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
Leaf-Lin added a commit to Leaf-Lin/elasticsearch that referenced this pull request Oct 17, 2022
* Add CCR limitation

closes elastic#86121

* Add restored index auto follow pattern restriction

elastic#87055

* Moving content to existing CCR page + several changes

* Remove sections to consolidate limitation information

* Delete separate file

* Remove restored indices from list of things that aren't replicated

Co-authored-by: Adam Locke <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
elasticsearchmachine pushed a commit that referenced this pull request Oct 18, 2022
* Add CCR limitation

closes #86121

* Add restored index auto follow pattern restriction

#87055

* Moving content to existing CCR page + several changes

* Remove sections to consolidate limitation information

* Delete separate file

* Remove restored indices from list of things that aren't replicated

Co-authored-by: Adam Locke <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>

Co-authored-by: Adam Locke <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
elasticsearchmachine pushed a commit that referenced this pull request Oct 18, 2022
* Add CCR limitation

closes #86121

* Add restored index auto follow pattern restriction

#87055

* Moving content to existing CCR page + several changes

* Remove sections to consolidate limitation information

* Delete separate file

* Remove restored indices from list of things that aren't replicated

Co-authored-by: Adam Locke <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>

Co-authored-by: Adam Locke <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
elasticsearchmachine pushed a commit that referenced this pull request Oct 18, 2022
* Add CCR limitation

closes #86121

* Add restored index auto follow pattern restriction

#87055

* Moving content to existing CCR page + several changes

* Remove sections to consolidate limitation information

* Delete separate file

* Remove restored indices from list of things that aren't replicated

Co-authored-by: Adam Locke <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>

Co-authored-by: Adam Locke <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
@csoulios csoulios added v8.5.0 and removed v8.5.1 labels Nov 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Indexing/CCR Issues around the Cross Cluster State Replication features >docs General docs changes Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. Team:Docs Meta label for docs team v8.3.4 v8.4.4 v8.5.0 v8.6.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[DOCS] Document Cross Cluster Replication (CCR) limitations
10 participants