Allow indices lookup to be built lazily #78745
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
by adding a flag to Metadata.Builder.build() method that controls when to build indices lookup. Relates to elastic#77466
martijnvg
force-pushed
the
lazily_built_indices_lookup2
branch
from
918d7a4 to
0ea5c7e
Compare
| @@ -189,7 +189,7 @@ | |||
| private final String[] allClosedIndices; | |||
| private final String[] visibleClosedIndices; | |||
|
|
|||
| private final SortedMap<String, IndexAbstraction> indicesLookup; | |||
| private volatile SortedMap<String, IndexAbstraction> indicesLookup; | |||
There was a problem hiding this comment.
No need for this to be volatile, this is just a best effort cache. No need to synchronize here. Especially when the only time this is null after construction is during the ILM task loop anyway.
| if (indicesLookup != null) { | ||
| return indicesLookup; | ||
| } | ||
| synchronized (this) { |
There was a problem hiding this comment.
Same here, let's not have any synchronization here. If this isn't set then it will be lazy set just fine in our task loops and otherwise it's set from the time of construction anyway.
| aliasToIndices.compute(aliasMetadata.getAlias(), (aliasName, indices) -> { | ||
| if (indices == null) { | ||
| indices = new ArrayList<>(); | ||
| aliasToIndices.compute(aliasMetadata.getAlias(), (aliasName, val) -> { |
There was a problem hiding this comment.
If we're changing this already (+1 to the name change btw), can't we just go for computeIfAbsent to make it look nicer? :) (and also faster because we don't need a capturing lambda then)
martijnvg
added
:Data Management/Indices APIs
|
Pinging @elastic/es-data-management (Team:Data Management) |
martijnvg
added
the
:Data Management/ILM+SLM
martijnvg
changed the title
original-brownbear
approved these changes
Oct 7, 2021
| return indicesLookup; | ||
| if (indicesLookup != null) { | ||
| return indicesLookup; | ||
| } else { |
There was a problem hiding this comment.
NIT: can unwrap the else I guess :)
| return build(true); | ||
| } | ||
|
|
||
| public Metadata build(boolean builtIndicesLookupEagerly) { |
There was a problem hiding this comment.
I think this one deserves a little Javadoc :)
97 tasks
💔 Backport failed
You can use sqren/backport to manually backport by running |
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this pull request
Oct 7, 2021
Backport elastic#78745 to 7.x branch. Overloaded the Metadata.Builder.build() method by adding a parameter that controls when to build indices lookup. By default the indices lookup is always build. When cluster state updates are batched in ilm, then built the indices lookup lazily. Because during batching only the final cluster state is used, then for the many intermediate cluster states we avoids building the indices lookup, which in a cluster with many indices can improve the performance of publishing cluster states significantly. Relates to elastic#77466
wjp719
added a commit
to wjp719/elasticsearch
that referenced
this pull request
Oct 7, 2021
…' into feature/data_stream_support_routing * wjp/feature/data_stream_support_routing: (44 commits) Revert "Adjust /_cat/templates not to request all metadata (elastic#78812)" Allow indices lookup to be built lazily (elastic#78745) [DOCS] Document default security in alpha2 (elastic#78227) Add cluster applier stats (elastic#77552) Fix failing URLDecodeProcessorTests::testProcessor test (elastic#78690) Upgrade to lucene snapshot ba75dc5e6bf (elastic#78817) Adjust /_cat/templates not to request all metadata (elastic#78812) Simplify build plugin license handling (elastic#77009) Fix SearchableSnapshotsBlobStoreCacheIntegTests.testBlobStoreCache (elastic#78616) Improve Docker image caching and testing (elastic#78552) Load knn vectors format with mmapfs (elastic#78724) Fix date math zone test to use negative minutes (elastic#78796) Changing name of shards field in node/stats api to shard_stats (elastic#78531) [DOCS] Fix system index refs in restore tutorial (elastic#78582) Add previously removed settings back for 8.0 (elastic#78784) TSDB: Fix template name in test Add a system property to forcibly format everything (elastic#78768) Revert "Adding config so that some tests will break if over-the-wire encryption fails (elastic#78409)" (elastic#78787) Must date math test failure Adding config so that some tests will break if over-the-wire encryption fails (elastic#78409) ...
elasticsearchmachine
pushed a commit
that referenced
this pull request
Oct 7, 2021
Backport #78745 to 7.x branch. Overloaded the Metadata.Builder.build() method by adding a parameter that controls when to build indices lookup. By default the indices lookup is always build. When cluster state updates are batched in ilm, then built the indices lookup lazily. Because during batching only the final cluster state is used, then for the many intermediate cluster states we avoids building the indices lookup, which in a cluster with many indices can improve the performance of publishing cluster states significantly. Relates to #77466
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overloaded the
Metadata.Builder.build()method by adding a parameter that controls when to build indices lookup.By default the indices lookup is always build.
When cluster state updates are batched in ilm, then built the indices lookup lazily. Because during batching only the final cluster state is used, then for the many intermediate cluster states we avoids building the indices lookup, which in a cluster with many indices can improve the performance of publishing cluster states significantly.
Relates to #77466