Add ability to disable importing dangling indices on shared filesystems

[dakrone]

We already have issues open for the behavior of importing indices when a node is out of the cluster while an index is deleted, however, in the case of shadow replicas on a shared filesystem, if the index is deleted while a node is not part of the cluster, when the node comes back it will try to re-import the index as dangling without any actual data, leading to exceptions like:

[2016-01-14 23:17:23,303][WARN ][indices.cluster ] [data1] [[my-index][2]] marking and sending shard failed due to [failed recovery]
[my-index][[my-index][2]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(default(mmapfs(/tmp/foo/my-index/2/index),niofs(/tmp/foo/my-index/2/index))): files: []];

It would be nice to have an option to disable this dangling import behavior only for indices on a shared filesystem, since this is likely to cause much more confusion than dangling imports on non-shared filesystems.

Related #13298

[bleskes]

Discussing the cause issue (index delete while node is off the cluster) I think we should consider this in a different way, which may help in the normal use case (non-shared FS) as well. Dangling indices are added as a safety guard against the case where a cluster goes down (or the master nodes) and because of mis configuration or because of bad operation practices (kill all masters, start fresh ones with a different data folders). In these cases the new master publishes a cluster state which is empty and we don't want to delete the data. These days we have a cluster UUID as part of the cluster meta data. That uuid is created once and never change in the duration of the cluster. That means that we can detect deletion while node left if:

1. Store the cluster uuid in the index meta data, so we know the cluster this index belongs to.
2. When a node gets a cluster state from a master and it doesn't contain the index it has on disk, the node can check whether the index used to be belong to the same cluster (by uuid) . If it is, we know it's deleted.
3. If the uuid in the index is different than the uuid in the cluster, we import it.

How does this sound?

[dakrone]

How does this sound?

I think this sounds great!