Backport support for replicating closed indices to 7.x #39506
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tlrx
added
:Distributed Indexing/Distributed
|
Pinging @elastic/es-distributed |
|
All tests passed:
@elasticmachine test this please |
|
All tests passed again:
@elasticmachine test this please |
|
One job failed with an unrelated issue (#28153):
@elasticmachine test this please |
|
All tests passed again:
@elasticmachine test this please |
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Mar 1, 2019
|
One job failed:
@elasticmachine test this please |
This commit adds a new NoOpEngine implementation based on the current ReadOnlyEngine. This new implementation uses an empty DirectoryReader with no segments readers and will always returns 0 docs. The NoOpEngine is the default Engine created for IndexShards of closed indices. It expects an empty translog when it is instantiated. Relates to elastic#33888
Changes were made in elastic#34357 and elastic#36467
When a NoOpEngine is instanciated, the current implementation verifies that the translog contains no operations and that it contains the same UUID as the last Lucene commit data.We can relax those two constraints because the Close Index API now ensure that all translog operations are flushed before closing a shard. The detection of coherence between translog UUID / Lucene commit data is not specific to NoOpEngine, and is already done by IndexShard.innerOpenEngineAndTranslog(). Related to elastic#33888
…astic#38024) This commit allows shards of indices in CLOSE state to be replicated as normal shards. It changes the MetaDataIndexStateService so that index routing tables of closed indices are kept in cluster state when the index is closed. Index routing tables are modified so that shard routings are reinitialized with the INDEX_CLOSED unassigned information. The IndicesClusterStateService is modified to remove IndexService instances of closed or reopened indices. In combination with the ShardRouting being in INITIALIZING state the shards are recreated on the data nodes to reflect the new state. If the index state is closed, the IndexShard instances will be created using the NoOpEngine as the engine implementation. This commit also mutes two tests that rely on the fact that shard locks are released when an index is closed, which is not the case anymore with replicated closed indices (actually the locks are released but reacquired once the shard is reinitialized after being closed). These tests will be adapted in follow up PRs. Finally, many things will require to be adapted or improved in follow up PRs (see elastic#33888) but this is the first big step towards replicated closed indices. Relates to elastic#33888
…dices (elastic#38329) Replicated closed indices do not need to be refreshed, neither they need their translogs or global checkpoint to be fsync. This pull request changes how `BaseAsyncTask` tasks are rescheduled in `IndexService` instances so that the tasks are rescheduled only when the index is opened. Relates to elastic#33888
…rs (elastic#38955) This pull request removes the legacy way of closing indices (aka "direct close") in mixed versions clusters, since this backward compatibility logic is not required anymore on master/8.0.0. It also changes the closing logic so that routing tables of closed indices are removed when the cluster contains a node in version < 8.0. Relates elastic#33888
…astic#38631) This pull request modifies the `ClusterAllocationExplainIT` test suite so that it always runs the tests with opened and closed indices. The only test that was not adapted for closed indices is `testAllocationFilteringOnIndexCreation` because we don't allow to directly create indices in the closed state. Relates to elastic#33888
This commit adds a simple test which verifies that a replica can be promoted as a primary when the index is closed. Relates to elastic#33888
* Adapt more tests suites to closed indices Similarly to elastic#38631, this pull request modifies multiple test suites so that they runs the tests with opened or closed indices. The suites are testing: - shard allocation filtering - shard allocation awereness - Reroute API Relates to elastic#33888
This commit adapts the Recovery API to make it work with shards of replicated closed indices. Relates elastic#33888
Closing an index is a process that can be broken down into several steps: 1. first, the state of the cluster is updated to add a write block on the index to be closed 2. then, a transport replication action is executed on all shards of the index. This action checks that the maximum sequence number and the global checkpoint have identical values, indicating that all in flight writing operations have been completed on the shard. 3. finally, and if the previous steps were successful, the cluster state is updated again to change the state of the index from `OPEN`to `CLOSE`. During the last step, the master node retrieves the minimum node version among all the nodes that compose the cluster: * If a node is in pre 8.0 version, the index is closed and the index routing table is removed from the cluster state. This is the "old" way of closing indices and closed indices with no routing table are not replicated. * If all nodes are in version 8.0 or higher, the index is closed and its routing table is reinitialized in cluster state. This is the new way of closing indices and such closed indices will be replicated in the cluster. But routing tables are not persisted in the cluster state, so after a full cluster restart there is no way to make the distinction between an index closed in 7.x and an index closed and replicated on 8.0. This commit introduces a new private index settings named `index.verified_before_close` that is added to closed indices that are replicated at closing time. This setting serves as a marker to indicate that the index has been closed using the new Close Index API on a cluster that supports replication of closed indices. This way, after a full cluster restart, the Gateway service can automatically recovers those closed indices as if they were opened indices. Closed indices that don't have this setting (because they were closed on a pre-8.0 cluster, or a cluster in mixed version) won't be recovered and will need to be reopened and closed again on a 8.0 cluster. Note that reopening the index removes the private setting. Relates to elastic#33888
Now the test `CloseFollowerIndexIT` has been added in elastic#38702, it needs to be adapted for replicated closed indices. The test closes the follower index which is lagging behind the leader index. When it's closed, no sanity checks are executed because it's a follower index (this is a consequence of elastic#38702). But with replicated closed indices, the index is reinitialized as a closed index with a `NoOpEngine` and such engines make strong assertions on the values of the maximum sequence number and the global checkpoint. Since the values do not match, the shards cannot be created and fail and the cluster health turns RED. This commit adapts the `CloseFollowerIndexIT` test so that it wraps the default `UncaughtExceptionHandler` with a handler that tolerates any exception thrown by `ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint()`. Replacing the default uncaught exception handler requires specific permissions, and instead of creating another gradle project it duplicates the `internalClusterTest` task to make it work without security manager for this specific test only. Relates to elastic#33888
This commit adapts the Cluster Health API to support replicated closed indices. In order to do that, it removes the hard coded indices options from the `ClusterHealthRequest` and replaces it with a new `IndicesOptions.lenientExpand()` option. This option will be used by the master node (once it is upgraded to 8.0) to compute the global cluster health using both opened and closed indices information by default. The `expand_wildcards` REST parameter is also documented and tests where added to ensure that a specific expansion type can be used to monitoring the health of a only opened or only closed indices. Since the Cat Indices relies on the Cluster Health API, it has been adapted to report information about closed indices too. Note that the health and number of shards/replicas is only printed out for closed indices that have an index routing table. Closed indices without routing table have the same output as before. Related to elastic#33888
This commit changes the Close Index API to add a `wait_for_active_shards` parameter that allows to wait for shards of closed indices to be active before returning a response. Relates elastic#33888
tlrx
force-pushed
the
replicated-closed-indices-7.x
branch
from
cd3d1c1 to
fa346c6
Compare
tlrx
added a commit
that referenced
this pull request
Mar 1, 2019
|
I've left one comment about BWC here: #39506 (comment) |
|
@ywelsch I've updated the code. Thanks for spotting my error :/ |
|
Thanks @ywelsch |
tlrx
added a commit
that referenced
this pull request
Mar 1, 2019
smalyshev
pushed a commit
to smalyshev/elasticsearch
that referenced
this pull request
Oct 23, 2024
…ed indices (elastic#114314) Support for replicating closed indices was added in elastic#39506 (7.1.0), we can expect the the cluster always supports replication of closed indices in 8.0/9.0
georgewallace
pushed a commit
to georgewallace/elasticsearch
that referenced
this pull request
Oct 25, 2024
…ed indices (elastic#114314) Support for replicating closed indices was added in elastic#39506 (7.1.0), we can expect the the cluster always supports replication of closed indices in 8.0/9.0
jfreden
pushed a commit
to jfreden/elasticsearch
that referenced
this pull request
Nov 4, 2024
…ed indices (elastic#114314) Support for replicating closed indices was added in elastic#39506 (7.1.0), we can expect the the cluster always supports replication of closed indices in 8.0/9.0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the backport of #39499 for
7.x.The change on master is not merged yet but I'm creating the backport PR to let it run few cycles on CI.