[Upgrade] migration system indices fails for tasks: mapper_parsing_exception

[liza-mae]

Elasticsearch Version

7.17.0

Installed Plugins

No response

Java Version

bundled

OS Version

cloud

Problem Description

When running automated upgrade testing, the upgrade assistant fails for tasks: mapper_parsing_exception.
Slack discussion: https://elastic.slack.com/archives/C5UDK4KDJ/p1644446831390349 cc: @gwbrown
Possibly also related to previous issue: #79681 and pr: #80054

Steps to Reproduce

Running upgrade path 5.16 -> 6.8.23 -> 7.17.0.

Logs (if relevant)

Response:

{"features":[{"indices":[{"failure_cause":{"error":{"reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [task : {_meta={version=7.17.0}, dynamic=strict, properties={task={properties={node={type=keyword}, headers={type=object, enabled=false}, parent_task_id={type=keyword}, running_time_in_nanos={type=long}, action={type=keyword}, description={type=text}, start_time_in_millis={type=long}, id={type=long}, type={type=keyword}, cancellable={type=boolean}, status={type=object, enabled=false}}}, response={type=object, enabled=false}, completed={type=boolean}, error={type=object, enabled=false}}}]","caused_by":{"reason":"Root mapping definition has unsupported parameters:  [task : {_meta={version=7.17.0}, dynamic=strict, properties={task={properties={node={type=keyword}, headers={type=object, enabled=false}, parent_task_id={type=keyword}, running_time_in_nanos={type=long}, action={type=keyword}, description={type=text}, start_time_in_millis={type=long}, id={type=long}, type={type=keyword}, cancellable={type=boolean}, status={type=object, enabled=false}}}, response={type=object, enabled=false}, completed={type=boolean}, error={type=object, enabled=false}}}]","stack_trace":"MapperParsingException[Root mapping definition has unsupported parameters:  [task : {_meta={version=7.17.0}, dynamic=strict, properties={task={properties={node={type=keyword}, headers={type=object, enabled=false}, parent_task_id={type=keyword}, running_time_in_nanos={type=long}, action={type=keyword}, description={type=text}, start_time_in_millis={type=long}, id={type=long}, type={type=keyword}, cancellable={type=boolean}, status={type=object, enabled=false}}}, response={type=object, enabled=false}, completed={type=boolean}, error={type=object, enabled=false}}}]]\n\tat org.elasticsearch.index.mapper.MappingParser.checkNoRemainingFields(MappingParser.java:72)\n\tat org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:145)\n\tat org.elasticsearch.index.mapper.MapperService.parseMappings(MapperService.java:503)\n\tat org.elasticsearch.index.mapper.MapperService.mergeAndApplyMappings(MapperService.java:419)\n\tat org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:402)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.updateIndexMappingsAndBuildSortOrder(MetadataCreateIndexService.java:1303)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.lambda$applyCreateIndexWithTemporaryService$3(MetadataCreateIndexService.java:453)\n\tat org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:673)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:451)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:567)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:413)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:420)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:319)\n\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:51)\n\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:836)\n\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:403)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:243)\n\tat org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\n","type":"mapper_parsing_exception"},"stack_trace":"MapperParsingException[Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [task : {_meta={version=7.17.0}, dynamic=strict, properties={task={properties={node={type=keyword}, headers={type=object, enabled=false}, parent_task_id={type=keyword}, running_time_in_nanos={type=long}, action={type=keyword}, description={type=text}, start_time_in_millis={type=long}, id={type=long}, type={type=keyword}, cancellable={type=boolean}, status={type=object, enabled=false}}}, response={type=object, enabled=false}, completed={type=boolean}, error={type=object, enabled=false}}}]]; nested: MapperParsingException[Root mapping definition has unsupported parameters:  [task : {_meta={version=7.17.0}, dynamic=strict, properties={task={properties={node={type=keyword}, headers={type=object, enabled=false}, parent_task_id={type=keyword}, running_time_in_nanos={type=long}, action={type=keyword}, description={type=text}, start_time_in_millis={type=long}, id={type=long}, type={type=keyword}, cancellable={type=boolean}, status={type=object, enabled=false}}}, response={type=object, enabled=false}, completed={type=boolean}, error={type=object, enabled=false}}}]];\n\tat org.elasticsearch.index.mapper.MapperService.parseMappings(MapperService.java:505)\n\tat org.elasticsearch.index.mapper.MapperService.mergeAndApplyMappings(MapperService.java:419)\n\tat org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:402)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.updateIndexMappingsAndBuildSortOrder(MetadataCreateIndexService.java:1303)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.lambda$applyCreateIndexWithTemporaryService$3(MetadataCreateIndexService.java:453)\n\tat org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:673)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:451)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:567)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:413)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:420)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:319)\n\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:51)\n\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:836)\n\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:403)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:243)\n\tat org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\nCaused by: MapperParsingException[Root mapping definition has unsupported parameters:  [task : {_meta={version=7.17.0}, dynamic=strict, properties={task={properties={node={type=keyword}, headers={type=object, enabled=false}, parent_task_id={type=keyword}, running_time_in_nanos={type=long}, action={type=keyword}, description={type=text}, start_time_in_millis={type=long}, id={type=long}, type={type=keyword}, cancellable={type=boolean}, status={type=object, enabled=false}}}, response={type=object, enabled=false}, completed={type=boolean}, error={type=object, enabled=false}}}]]\n\tat org.elasticsearch.index.mapper.MappingParser.checkNoRemainingFields(MappingParser.java:72)\n\tat org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:145)\n\tat org.elasticsearch.index.mapper.MapperService.parseMappings(MapperService.java:503)\n\t... 24 more\n","type":"mapper_parsing_exception","root_cause":[{"reason":"Root mapping definition has unsupported parameters:  [task : {_meta={version=7.17.0}, dynamic=strict, properties={task={properties={node={type=keyword}, headers={type=object, enabled=false}, parent_task_id={type=keyword}, running_time_in_nanos={type=long}, action={type=keyword}, description={type=text}, start_time_in_millis={type=long}, id={type=long}, type={type=keyword}, cancellable={type=boolean}, status={type=object, enabled=false}}}, response={type=object, enabled=false}, completed={type=boolean}, error={type=object, enabled=false}}}]","stack_trace":"MapperParsingException[Root mapping definition has unsupported parameters:  [task : {_meta={version=7.17.0}, dynamic=strict, properties={task={properties={node={type=keyword}, headers={type=object, enabled=false}, parent_task_id={type=keyword}, running_time_in_nanos={type=long}, action={type=keyword}, description={type=text}, start_time_in_millis={type=long}, id={type=long}, type={type=keyword}, cancellable={type=boolean}, status={type=object, enabled=false}}}, response={type=object, enabled=false}, completed={type=boolean}, error={type=object, enabled=false}}}]]\n\tat org.elasticsearch.index.mapper.MappingParser.checkNoRemainingFields(MappingParser.java:72)\n\tat org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:145)\n\tat org.elasticsearch.index.mapper.MapperService.parseMappings(MapperService.java:503)\n\tat org.elasticsearch.index.mapper.MapperService.mergeAndApplyMappings(MapperService.java:419)\n\tat org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:402)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.updateIndexMappingsAndBuildSortOrder(MetadataCreateIndexService.java:1303)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.lambda$applyCreateIndexWithTemporaryService$3(MetadataCreateIndexService.java:453)\n\tat org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:673)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:451)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:567)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:413)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:420)\n\tat org.elasticsearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:319)\n\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:51)\n\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:836)\n\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:403)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:243)\n\tat org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:833)\n","type":"mapper_parsing_exception"}]}},"index":".tasks","version":"6.8.23"}],"feature_name":"tasks","migration_status":"ERROR","minimum_index_version":"6.8.23"}],"migration_status":"ERROR"}

Output from get .tasks

#! this request accesses system indices: [.tasks], but in a future major version, direct access to system indices will be prevented by default
{
  ".tasks" : {
    "aliases" : { },
    "mappings" : {
      "dynamic" : "strict",
      "_meta" : {
        "version" : "7.17.0"
      },
      "properties" : {
        "completed" : {
          "type" : "boolean"
        },
        "error" : {
          "type" : "object",
          "enabled" : false
        },
        "response" : {
          "type" : "object",
          "enabled" : false
        },
        "task" : {
          "properties" : {
            "action" : {
              "type" : "keyword"
            },
            "cancellable" : {
              "type" : "boolean"
            },
            "description" : {
              "type" : "text"
            },
            "headers" : {
              "type" : "object",
              "enabled" : false
            },
            "id" : {
              "type" : "long"
            },
            "node" : {
              "type" : "keyword"
            },
            "parent_id" : {
              "type" : "keyword"
            },
            "parent_task_id" : {
              "type" : "keyword"
            },
            "running_time_in_nanos" : {
              "type" : "long"
            },
            "start_time_in_millis" : {
              "type" : "long"
            },
            "status" : {
              "type" : "object",
              "enabled" : false
            },
            "type" : {
              "type" : "keyword"
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "blocks" : {
          "read_only_allow_delete" : "false"
        },
        "provided_name" : ".tasks",
        "creation_date" : "1644442928199",
        "priority" : "2147483647",
        "number_of_replicas" : "0",
        "uuid" : "xT6lagX9QA27aW-EbTLMJw",
        "version" : {
          "created" : "6082399"
        }
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[gwbrown]

I dug into this today and figured out what's going on.

It's a combo of the tasks index having some special handling to deal with it still using a named type in its canonical mappings in 7.17, which wasn't properly handled in the migration code. This will be a very straightforward fix, fortunately - I'll have a PR up shortly.

As far as a workaround until we get a patch out, the .tasks index is just historical data for troubleshooting purposes so it can be safely deleted - and in fact that's what we recommended for the 6->7 major version upgrade, simply deleting the .tasks index due to complications in reindexing it (which we were able to avoid this version, but it appears we didn't test it thoroughly enough).

[gwbrown]

PR to fix is up at #83817.

[gwbrown]

This should be fixed in 7.17.1 with #83817, so I'm going to close this issue.