Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replicate closed indices #33888

Closed
50 tasks done
tlrx opened this issue Sep 20, 2018 · 4 comments
Closed
50 tasks done

Replicate closed indices #33888

tlrx opened this issue Sep 20, 2018 · 4 comments
Assignees
@tlrx @ywelsch @dnhatn
Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >feature Meta v7.2.0

Comments

@tlrx
Copy link
Member

tlrx commented Sep 20, 2018 •
edited by dnhatn
Loading

Closed indices currently have the downside that they are not replicated, so it's possible for an index to be closed, then the data lost because the node was terminated without saving the data.

We'd like to be able to close indices to remove any memory overhead of them, but still have the indices available for replication.

Steps to accomplish this (not necessarily in order):

Routing Table

Open/Close Index APIs

Engine/Translog

Tests

Others
@tlrx tlrx added >feature :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. labels Sep 20, 2018
@tlrx tlrx self-assigned this Sep 20, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@tlrx tlrx mentioned this issue Sep 20, 2018
Closed
8 tasks
tlrx added a commit that referenced this issue Sep 26, 2018
@tlrx
[RCI] Add NoOpEngine for closed indices (#33903)
d2be022 
This commit adds a new NoOpEngine implementation based on the current 
ReadOnlyEngine. This new implementation uses an empty DirectoryReader 
with no segments readers and will always returns 0 docs. The NoOpEngine 
is the default Engine created for IndexShards of closed indices. It expects 
an empty translog when it is instantiated.

Relates to #33888
@tlrx tlrx mentioned this issue Sep 27, 2018
Closed
tlrx added a commit that referenced this issue Oct 26, 2018
@tlrx
[RCI] Add NoOpEngine for closed indices (#33903)
8e93bc4 
This commit adds a new NoOpEngine implementation based on the current 
ReadOnlyEngine. This new implementation uses an empty DirectoryReader 
with no segments readers and will always returns 0 docs. The NoOpEngine 
is the default Engine created for IndexShards of closed indices. It expects 
an empty translog when it is instantiated.

Relates to #33888
@tlrx tlrx mentioned this issue Oct 26, 2018
Merged
@tlrx tlrx mentioned this issue Nov 7, 2018
Merged
tlrx added a commit to tlrx/elasticsearch that referenced this issue Nov 7, 2018
@tlrx
[RCI] Add IndexShardOperationPermits.asyncBlockOperations(ActionListe…
4d792af 
…ner<Releasable>)

The current implementation of asyncBlockOperations() can be used to
execute some code once all indexing operations permits have been acquired,
 then releases all permits immediately after the code execution. This
 immediate release is not suitable for treatments that need to keep all
 permits over multiple execution steps.

This commit adds a new asyncBlockOperations() that exposes a Releasable,
 making it possible to acquire all permits and only release them all
 when needed by closing the Releasable.

This method is aimed to be used in a TransportReplicationAction that
 will acquire all permits on the primary shard.

The existing blockOperations() and asyncBlockOperations() methods have
been modified to delegate permit acquisition/releasing to this new
method.

Relates to elastic#33888
tlrx added a commit that referenced this issue Nov 8, 2018
@tlrx
[RCI] Add IndexShardOperationPermits.asyncBlockOperations(ActionListe…
1703a61 
…ner<Releasable>) (#34902)

The current implementation of asyncBlockOperations() can be used to
execute some code once all indexing operations permits have been acquired,
 then releases all permits immediately after the code execution. This
 immediate release is not suitable for treatments that need to keep all
 permits over multiple execution steps.

This commit adds a new asyncBlockOperations() that exposes a Releasable,
 making it possible to acquire all permits and only release them all
 when needed by closing the Releasable. The existing blockOperations() 
method has been modified to delegate permit acquisition/releasing to this new
method.

Relates to #33888
tlrx added a commit that referenced this issue Nov 8, 2018
@tlrx
[RCI] Add NoOpEngine for closed indices (#33903)
95cfc1e 
This commit adds a new NoOpEngine implementation based on the current 
ReadOnlyEngine. This new implementation uses an empty DirectoryReader 
with no segments readers and will always returns 0 docs. The NoOpEngine 
is the default Engine created for IndexShards of closed indices. It expects 
an empty translog when it is instantiated.

Relates to #33888
tlrx added a commit that referenced this issue Nov 8, 2018
@tlrx
[RCI] Add NoOpEngine for closed indices (#33903)
c25c9bf 
This commit adds a new NoOpEngine implementation based on the current 
ReadOnlyEngine. This new implementation uses an empty DirectoryReader 
with no segments readers and will always returns 0 docs. The NoOpEngine 
is the default Engine created for IndexShards of closed indices. It expects 
an empty translog when it is instantiated.

Relates to #33888
@s1monw s1monw mentioned this issue Nov 9, 2018
Closed
@ywelsch ywelsch added the Meta label Nov 12, 2018
pgomulka pushed a commit to pgomulka/elasticsearch that referenced this issue Nov 13, 2018
@tlrx @pgomulka
[RCI] Add IndexShardOperationPermits.asyncBlockOperations(ActionListe…
e8fb974 
…ner<Releasable>) (elastic#34902)

The current implementation of asyncBlockOperations() can be used to
execute some code once all indexing operations permits have been acquired,
 then releases all permits immediately after the code execution. This
 immediate release is not suitable for treatments that need to keep all
 permits over multiple execution steps.

This commit adds a new asyncBlockOperations() that exposes a Releasable,
 making it possible to acquire all permits and only release them all
 when needed by closing the Releasable. The existing blockOperations() 
method has been modified to delegate permit acquisition/releasing to this new
method.

Relates to elastic#33888
tlrx added a commit that referenced this issue Nov 14, 2018
@tlrx
[RCI] Check blocks while having index shard permit in TransportReplic…
31567ce 
…ationAction (#35332)

Today, the TransportReplicationAction checks the global level blocks and 
the index level blocks before routing the operation to the primary, in the 
ReroutePhase, and it happens at the very beginning of the transport 
replication action execution. For the upcoming rework of the Close Index 
API and in order to deal with primary relocation, we'll need to also check 
for blocks before executing the operation on the primary (while holding a 
permit) but before routing to the new primary.

This pull request change the AsyncPrimaryAction so that it checks for 
replication action's blocks before executing the operation locally or before 
routing the primary action to the newly primary shard. The check is done 
while holding a PrimaryShardReference.

Related to #33888
tlrx added a commit that referenced this issue Nov 14, 2018
@tlrx
[RCI] Add IndexShardOperationPermits.asyncBlockOperations(ActionListe…
d2f2505 
…ner<Releasable>) (#34902)

The current implementation of asyncBlockOperations() can be used to
execute some code once all indexing operations permits have been acquired,
 then releases all permits immediately after the code execution. This
 immediate release is not suitable for treatments that need to keep all
 permits over multiple execution steps.

This commit adds a new asyncBlockOperations() that exposes a Releasable,
 making it possible to acquire all permits and only release them all
 when needed by closing the Releasable. The existing blockOperations() 
method has been modified to delegate permit acquisition/releasing to this new
method.

Relates to #33888
tlrx added a commit that referenced this issue Nov 14, 2018
@tlrx
[RCI] Check blocks while having index shard permit in TransportReplic…
0c5e87f 
…ationAction (#35332)

Today, the TransportReplicationAction checks the global level blocks and
the index level blocks before routing the operation to the primary, in the
ReroutePhase, and it happens at the very beginning of the transport
replication action execution. For the upcoming rework of the Close Index
API and in order to deal with primary relocation, we'll need to also check
for blocks before executing the operation on the primary (while holding a
permit) but before routing to the new primary.

This pull request change the AsyncPrimaryAction so that it checks for
replication action's blocks before executing the operation locally or before
routing the primary action to the newly primary shard. The check is done
while holding a PrimaryShardReference.

Related to #33888
tlrx added a commit to tlrx/elasticsearch that referenced this issue Jun 25, 2019
@tlrx
Fix indices shown in _cat/indices (elastic#43286)
21e377d 
After two recent changes (elastic#38824 and elastic#33888), the _cat/indices API
no longer report information for active recovering indices and
non-replicated closed indices. It also misreport replicated closed
indices that are potentially not authorized for the user.

This commit changes how the cat action works by first using the
Get Settings API in order to resolve authorized indices. It then uses
the Cluster State, Cluster Health and Indices Stats APIs to retrieve
 information about the indices.

Closes elastic#39933
@tlrx tlrx mentioned this issue Jun 25, 2019
Merged
tlrx added a commit that referenced this issue Jun 25, 2019
@tlrx
Fix indices shown in _cat/indices (#43286)
0dc1c12 
After two recent changes (#38824 and #33888), the _cat/indices API
no longer report information for active recovering indices and
non-replicated closed indices. It also misreport replicated closed
indices that are potentially not authorized for the user.

This commit changes how the cat action works by first using the
Get Settings API in order to resolve authorized indices. It then uses
the Cluster State, Cluster Health and Indices Stats APIs to retrieve
 information about the indices.

Closes #39933
tlrx added a commit to tlrx/elasticsearch that referenced this issue Jun 26, 2019
@tlrx
Fix indices shown in _cat/indices (elastic#43286)
af5c394 
After two recent changes (elastic#38824 and elastic#33888), the _cat/indices API
no longer report information for active recovering indices and
non-replicated closed indices. It also misreport replicated closed
indices that are potentially not authorized for the user.

This commit changes how the cat action works by first using the
Get Settings API in order to resolve authorized indices. It then uses
the Cluster State, Cluster Health and Indices Stats APIs to retrieve
information about the indices.

Closes elastic#39933
@tlrx tlrx mentioned this issue Jun 26, 2019
Merged
tlrx added a commit that referenced this issue Jun 26, 2019
@tlrx
Fix indices shown in _cat/indices (#43286)
9aa340c 
After two recent changes (#38824 and #33888), the _cat/indices API
no longer report information for active recovering indices and
non-replicated closed indices. It also misreport replicated closed
indices that are potentially not authorized for the user.

This commit changes how the cat action works by first using the
Get Settings API in order to resolve authorized indices. It then uses
the Cluster State, Cluster Health and Indices Stats APIs to retrieve
information about the indices.

Closes #39933
@DaveCTurner DaveCTurner mentioned this issue Jul 26, 2019
Closed
@codebrain codebrain mentioned this issue Aug 5, 2019
Closed
37 tasks
@Mpdreamz Mpdreamz mentioned this issue Aug 7, 2019
Closed
16 tasks
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Jan 6, 2020
@DaveCTurner
Collect shard sizes for closed indices
019fda6 
Today the `InternalClusterInfoService` collects information on the sizes of
shards of open indices, but does not consider closed indices. This means that
shards of closed indices are treated as having zero size when they are being
allocated. This commit fixes this, obtaining the sizes of all shards.

Relates elastic#33888
@DaveCTurner DaveCTurner mentioned this issue Jan 6, 2020
Merged
DaveCTurner added a commit that referenced this issue Jan 6, 2020
@DaveCTurner
Collect shard sizes for closed indices (#50645)
604172e 
Today the `InternalClusterInfoService` collects information on the sizes of
shards of open indices, but does not consider closed indices. This means that
shards of closed indices are treated as having zero size when they are being
allocated. This commit fixes this, obtaining the sizes of all shards.

Relates #33888
DaveCTurner added a commit that referenced this issue Jan 6, 2020
@DaveCTurner
Collect shard sizes for closed indices (#50645)
66c6909 
Today the `InternalClusterInfoService` collects information on the sizes of
shards of open indices, but does not consider closed indices. This means that
shards of closed indices are treated as having zero size when they are being
allocated. This commit fixes this, obtaining the sizes of all shards.

Relates #33888
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this issue Jan 23, 2020
@DaveCTurner @SivagurunathanV
Collect shard sizes for closed indices (elastic#50645)
3e449ac 
Today the `InternalClusterInfoService` collects information on the sizes of
shards of open indices, but does not consider closed indices. This means that
shards of closed indices are treated as having zero size when they are being
allocated. This commit fixes this, obtaining the sizes of all shards.

Relates elastic#33888
This was referenced Feb 3, 2020
Closed
Closed
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Aug 19, 2020
@DaveCTurner
Add note on the danger of closed indices
7796e7e 
Prior to elastic#33888 it is dangerous to keep closed indices in your cluster
long-term: Elasticsearch does not maintain their shard copies so they
tend to get lost as the cluster migrates to new nodes. This risk isn't
documented today. This commit addresses that gap.
@DaveCTurner DaveCTurner mentioned this issue Aug 19, 2020
Merged
DaveCTurner added a commit that referenced this issue Aug 20, 2020
@DaveCTurner
Add note on the danger of closed indices (#61332)
37825c3 
Prior to #33888 it is dangerous to keep closed indices in your cluster
long-term: Elasticsearch does not maintain their shard copies so they
tend to get lost as the cluster migrates to new nodes. This risk isn't
documented today. This commit addresses that gap.
@mark54g mark54g mentioned this issue Sep 7, 2020
Open
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Dec 18, 2020
@DaveCTurner
Add ?wait_for_active_shards=default
169f49f 
Today you cannot explicitly indicate that an operation should use the
usual behaviour of waiting for active shards according to the underlying
index setting. This is a problem for the close index API which has a
default of `none` in 7.x for BWC reasons (see elastic#33888), but the usual
behaviour in 8.0: you cannot today opt-in to the 8.0 behaviour with this
parameter.

This commit adds support for the literal value `default` for the
`wait_for_active_shards` query parameter.

Relates elastic#66419
@DaveCTurner DaveCTurner mentioned this issue Dec 18, 2020
Closed
@DaveCTurner DaveCTurner mentioned this issue Jan 17, 2022
Merged
@DaveCTurner DaveCTurner mentioned this issue Nov 21, 2022
Closed
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Nov 21, 2022
@DaveCTurner
Skip ancient closed indices in desired balance
72b3829 
This assertion fails in the presence of pre-7.2.0 closed indices because
such indices don't even have routing table entries.

Relates elastic#33888
Closes elastic#91470
@DaveCTurner DaveCTurner mentioned this issue Nov 21, 2022
Merged
elasticsearchmachine pushed a commit that referenced this issue Nov 21, 2022
@DaveCTurner
Skip ancient closed indices in desired balance (#91765)
1c9a39c 
This assertion fails in the presence of pre-7.2.0 closed indices because
such indices don't even have routing table entries.

Relates #33888 Closes #91470
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Nov 21, 2022
@DaveCTurner
Skip ancient closed indices in desired balance (elastic#91765)
7470e52 
This assertion fails in the presence of pre-7.2.0 closed indices because
such indices don't even have routing table entries.

Relates elastic#33888 Closes elastic#91470
elasticsearchmachine pushed a commit that referenced this issue Nov 21, 2022
@DaveCTurner
Skip ancient closed indices in desired balance (#91765) (#91768)
166247f 
This assertion fails in the presence of pre-7.2.0 closed indices because
such indices don't even have routing table entries.

Relates #33888 Closes #91470
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tlrx tlrx

@ywelsch ywelsch

@dnhatn dnhatn

Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >feature Meta v7.2.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@colings86 @tlrx @ywelsch @dnhatn @elasticmachine