Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IndexMetadataUpdater#applyChanges is rather inefficient #78980

Closed
Tracked by #77466
original-brownbear opened this issue Oct 12, 2021 · 1 comment · Fixed by #79004
Closed
Tracked by #77466

IndexMetadataUpdater#applyChanges is rather inefficient #78980

original-brownbear opened this issue Oct 12, 2021 · 1 comment · Fixed by #79004
Assignees
@martijnvg
Labels
>bug :Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

Comments

@original-brownbear
Copy link
Member

Same as #77888

image

slightly smaller impact than #77888 but in the same ballpark. This method just adjusts allocation ids and primary terms it seems, so none of the information relevant to computing the indices lookup changes here => we should find a way to not have to recompute that to make reroutes a little faster.

relates #77466
@original-brownbear original-brownbear added >bug :Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. labels Oct 12, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Oct 12, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@original-brownbear original-brownbear mentioned this issue Oct 12, 2021
Open
97 tasks
@martijnvg martijnvg mentioned this issue Oct 12, 2021
Merged
martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Oct 15, 2021
@martijnvg
Reuse previous indices lookup in IndexMetadataUpdater#applyChanges(...)
feb7268 
The IndexMetadataUpdater#applyChanges(...) method builds a new metadata
instance, but only primary term or insync allocations may be updated.
No new indices, aliases or data streams are added, so re-building indices
lookup is not necessary.

In clusters with many indices the cost of building indices lookup is
non-neglectable and should be avoided in this case.

Closes elastic#78980
martijnvg added a commit that referenced this issue Oct 26, 2021
@martijnvg
Reuse previous indices lookup when possible (#79004)
bada1ba 
In cases when indices, aliases and data streams aren't modified then
the indices lookup can be reused.

For example in:
* The IndexMetadataUpdater#applyChanges(...) method builds a new metadata
instance, but only primary term or insync allocations may be updated.
No new indices, aliases or data streams are added, so re-building indices
lookup is not necessary.
* MasterService#patchVersions

Additionally the logic that checks when indices lookup can be reused,
this logic also checks the hidden and system flags of indices/datastreams.

In clusters with many indices the cost of building indices lookup is
non-neglectable and should be avoided in this case.

Closes #78980
Partially addresses to #77888
martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Oct 26, 2021
@martijnvg
Reuse previous indices lookup when possible
9f8708c 
Backporting elastic#79004 to 7.x branch.

In cases when indices, aliases and data streams aren't modified then
the indices lookup can be reused.

For example in:
* The IndexMetadataUpdater#applyChanges(...) method builds a new metadata
instance, but only primary term or insync allocations may be updated.
No new indices, aliases or data streams are added, so re-building indices
lookup is not necessary.
* MasterService#patchVersions

Additionally the logic that checks when indices lookup can be reused,
this logic also checks the hidden and system flags of indices/datastreams.

In clusters with many indices the cost of building indices lookup is
non-neglectable and should be avoided in this case.

Closes elastic#78980
Partially addresses elastic#77888
@martijnvg martijnvg mentioned this issue Oct 26, 2021
Merged
martijnvg added a commit that referenced this issue Oct 26, 2021
@martijnvg
Reuse previous indices lookup when possible (#79804)
7e9b8cd 
Backporting #79004 to 7.16 branch.

In cases when indices, aliases and data streams aren't modified then
the indices lookup can be reused.

For example in:
* The IndexMetadataUpdater#applyChanges(...) method builds a new metadata
instance, but only primary term or insync allocations may be updated.
No new indices, aliases or data streams are added, so re-building indices
lookup is not necessary.
* MasterService#patchVersions

Additionally the logic that checks when indices lookup can be reused,
this logic also checks the hidden and system flags of indices/datastreams.

In clusters with many indices the cost of building indices lookup is
non-neglectable and should be avoided in this case.

Closes #78980
Partially addresses #77888
lockewritesdocs pushed a commit to lockewritesdocs/elasticsearch that referenced this issue Oct 28, 2021
@martijnvg
Reuse previous indices lookup when possible (elastic#79004)
a2c3298 
In cases when indices, aliases and data streams aren't modified then
the indices lookup can be reused.

For example in:
* The IndexMetadataUpdater#applyChanges(...) method builds a new metadata
instance, but only primary term or insync allocations may be updated.
No new indices, aliases or data streams are added, so re-building indices
lookup is not necessary.
* MasterService#patchVersions

Additionally the logic that checks when indices lookup can be reused,
this logic also checks the hidden and system flags of indices/datastreams.

In clusters with many indices the cost of building indices lookup is
non-neglectable and should be avoided in this case.

Closes elastic#78980
Partially addresses to elastic#77888
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@martijnvg martijnvg

Labels
>bug :Distributed Coordination/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Reuse previous indices lookup when possible martijnvg/elasticsearch
3 participants
@martijnvg @original-brownbear @elasticmachine