Index deletes not applied when cluster UUID has changed #16825
Conversation
abeyad
force-pushed
the
bug-concurrent-index-del-reimports
branch
4 times, most recently
from
dddd41b to
dddf9de
Compare
| @@ -152,46 +168,35 @@ public boolean indexMetaDataChanged(IndexMetaData current) { | |||
| return true; | |||
| } | |||
|
|
|||
| public boolean blocksChanged() { | |||
There was a problem hiding this comment.
Please don't remove this method here, it would make this a breaking change in that it would break any plugins that relied on watching for ClusterChangeEvents where the blocks have changed
abeyad
force-pushed
the
bug-concurrent-index-del-reimports
branch
from
dddf9de to
011ac55
Compare
| @@ -134,10 +142,18 @@ public boolean indexRoutingTableChanged(String index) { | |||
| return deleted == null ? Collections.<String>emptyList() : deleted; | |||
| } | |||
|
|
|||
| /** | |||
| * Returns <code>true</code> iff the metadata for the cluster has changed between | |||
There was a problem hiding this comment.
can we add a note saying this is an reference level equality and not a true equal.
There was a problem hiding this comment.
Done, I also added that note to every other reference equality check in the class
|
I left some very minor comments. Looks good. I thought we had a test somewhere about index deletion with master change (which we should be able to enable now), but maybe I'm wrong. @brwe do you remember? |
|
also re the pr description:
This inaccurate - on disk dangling indices will still be imported - for example if a node was off while the index was deleted. This change only deals with the case where a node was isolated from the cluster while a delete happened (but was up!) and where a master failure happened during a delete operation (but after the delete was already committed). I think we should something like: If a node was isloated from the cluster while a delete was happening, the node will ignore the deleted operation when rejoining as we couldn't detect whether the new master genuinely deleted the indices or is a new fresh "resetted" master that was started without the old data folder. We can now be smarter and detect these resetted masters and actually delete the indices if this is not the case. Note that this new protection doesn't hold if the node was shut down. In that case it's indices will still be imported as dangling indices. |
I think that is the one that @abeyad enabled here: https://github.com/elastic/elasticsearch/pull/16825/files#diff-c454159f60a0127854028ccb5502500eL1076 I know of no other one. |
|
@brwe I missed that one line in the white space fixing. Jet lag... :( thanks! |
|
LGTM. Thanks @abeyad |
If a node was isolated from the cluster while a delete was happening, the node will ignore the deleted operation when rejoining as we couldn't detect whether the new master genuinely deleted the indices or it is a new fresh "reset" master that was started without the old data folder. We can now be smarter and detect these reset masters and actually delete the indices on the node if its not the case of a reset master. Note that this new protection doesn't hold if the node was shut down. In that case it's indices will still be imported as dangling indices. Closes elastic#11665
abeyad
force-pushed
the
bug-concurrent-index-del-reimports
branch
from
67dd695 to
45ae99c
Compare
This commit backports commit 83d1e09 from master to 2.x. Relates elastic#16825
clintongormley
added
:Distributed Indexing/Distributed
If a node was isolated from the cluster while a delete was happening, the node will ignore the deleted operation when rejoining as we couldn't detect whether the new master genuinely deleted the indices or it is a new fresh "reset" master that was started without the old data folder. We can now be smarter and detect these reset masters and actually delete the indices on the node if its not the case of a reset master.
Note that this new protection doesn't hold if the node was shut down. In that case it's indices will still be imported as dangling indices.
Closes #11665