snapshot succeeds with missing index defined

[jpcarey]

Elasticsearch version: 5.4.1

Plugins installed: x-pack

JVM version:
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

OS version:
16.6.0 Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:
Creating a snapshot where one of the indices does not exist should result in a failed snapshot unless you have ignore_unavailable set to true. Rather, the snapshot succeeds and there is no indication of the missing index.

PUT /_snapshot/my_backup
{
  "type": "fs",
  "settings": {
    "compress": true,
    "location": "/Users/jared/tmp/blah"
  }
}

POST exists/test/1
{"hello": true}

PUT /_snapshot/my_backup/snapshot_1
{"indices":"exists,missing"}

GET /_snapshot/my_backup/snapshot_1/_status

# return
{
  "snapshots": [
    {
      "snapshot": "snapshot_1",
      "repository": "my_backup",
      "uuid": "Zp8WS3YdST-c65Ku_r9sVA",
      "state": "SUCCESS",
      "shards_stats": {
        "initializing": 0,
        "started": 0,
        "finalizing": 0,
        "done": 5,
        "failed": 0,
        "total": 5
      },
      "stats": {
        "number_of_files": 8,
        "processed_files": 8,
        "total_size_in_bytes": 3203,
        "processed_size_in_bytes": 3203,
        "start_time_in_millis": 1498150300101,
        "time_in_millis": 36
      },
      "indices": {
        "exists": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 5,
            "failed": 0,
            "total": 5
          },
          "stats": {
            "number_of_files": 8,
            "processed_files": 8,
            "total_size_in_bytes": 3203,
            "processed_size_in_bytes": 3203,
            "start_time_in_millis": 1498150300101,
            "time_in_millis": 36
          },
          "shards": {
            ... 
          }
        }
      }
    }
  ]
}

[PnPie]

Hello,

I've had a look at the index name resolver in TransportCreateSnapshotAction and it seems it works well. What is very strange is that I created a test function in SharedClusterSnapshotRestoreIT in which I just passed a missing index for the creation of a snapshot:

public void testSnapshotOnMissingIndex() throws Exception {
    final Client client = client();

    logger.info("--> creating repository");
    assertAcked(client.admin().cluster().preparePutRepository("test-repo")
                .setType("fs").setSettings(Settings.builder().put("location", randomRepoPath())));

    logger.info("--> snapshot");
    client.admin().cluster()
        .prepareCreateSnapshot("test-repo", "test-snap")
        .setIndices("missing")
        .get();
}

And it does throw an IndexNotFoundException:

[missing] IndexNotFoundException[no such index]
	at __randomizedtesting.SeedInfo.seed([3D91A5BCCC7B5F6C:9E614A94BC995C9B]:0)
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:664)
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:626)
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:583)
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:162)
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:141)
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:74)
	at org.elasticsearch.action.admin.cluster.snapshots.create.TransportCreateSnapshotAction.checkBlock(TransportCreateSnapshotAction.java:69)
	at org.elasticsearch.action.admin.cluster.snapshots.create.TransportCreateSnapshotAction.checkBlock(TransportCreateSnapshotAction.java:1)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:135)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:127)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:105)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:1)
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:1)
	at org.elasticsearch.transport.AssertingTransportInterceptor$1.messageReceived(AssertingTransportInterceptor.java:76)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1527)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:139)
	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1484)
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1354)
	at org.elasticsearch.transport.MockTcpTransport.readMessage(MockTcpTransport.java:170)
	at org.elasticsearch.transport.MockTcpTransport.access$7(MockTcpTransport.java:148)
	at org.elasticsearch.transport.MockTcpTransport$MockChannel$1.lambda$0(MockTcpTransport.java:348)
	at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105)
	at org.elasticsearch.transport.MockTcpTransport$MockChannel$1.doRun(MockTcpTransport.java:348)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

But when I test it on Elasticsearch it was passed with no error message and the missing index is just to be ignored as above. I don't really understand because in the Rest layer we don't do any index resolving and the indices should be passed as it is to create a CreateSnapshotRequest ? So where the problem may lie ?

[liketic]

I've made some investigation, seems like the default value of ignore_unavailable is true. Need more effort to figure it out. @abeyad I'm new to es, could you please provide some hints? Thanks in advance. 😄