Updating index metadata does an expensive validation of the mapping even if unchanged

[DaveCTurner]

Updating index settings on a large number of indices can take many minutes and the resulting cluster state update might fail to cleanly publish without warning. A significant part of this slowness is that the index metadata validation that is run for each index. This validation deserialises + reserializes the mapping for every index that got updated, which for large mappings combined with a large number of updates indices can take many minutes.

  100.0% [cpu=98.6%, other=1.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elasticsearch-5][masterService#updateTask][T#1]'
     10/10 snapshots sharing following 26 elements
       app/[email protected]/org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:150)
       app/[email protected]/org.elasticsearch.index.mapper.ObjectMapper$Builder.build(ObjectMapper.java:171)
       app/[email protected]/org.elasticsearch.index.mapper.ObjectMapper$Builder.build(ObjectMapper.java:64)
       app/[email protected]/org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:150)
       app/[email protected]/org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:110)
       app/[email protected]/org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:99)
       app/[email protected]/org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94)
       app/[email protected]/org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:370)
       app/[email protected]/org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347)
       app/[email protected]/org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337)
       app/[email protected]/org.elasticsearch.indices.IndicesService.verifyIndexMetadata(IndicesService.java:810)
       app/[email protected]/org.elasticsearch.cluster.metadata.MetadataUpdateSettingsService$1.execute(MetadataUpdateSettingsService.java:247)
       app/[email protected]/org.elasticsearch.cluster.metadata.MetadataUpdateSettingsService.lambda$new$0(MetadataUpdateSettingsService.java:79)
       app/[email protected]/org.elasticsearch.cluster.metadata.MetadataUpdateSettingsService$$Lambda$3405/0x00000008014c4400.execute(Unknown Source)
       app/[email protected]/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:908)
       app/[email protected]/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:878)
       app/[email protected]/org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:248)
       app/[email protected]/org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:156)
       app/[email protected]/org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:110)
       app/[email protected]/org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:148)
       app/[email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:709)
       app/[email protected]/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:260)
       app/[email protected]/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:223)
       [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       [email protected]/java.lang.Thread.run(Thread.java:833)

We should find a way to skip unnecessary mapping validation when nothing about the mappings has changed, and to make use of mapping deduplication here too.

Relates #77466
Extracted from #87120

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[romseygeek]

I think we can do this relatively simply by replacing the call to MapperService.merge() with a call to MapperService.updateMapping(), which already has some sanity checks to ensure that we don't needlessly apply updates when the mapping hasn't actually changed.

[javanna]

Alan tried to fix this but realized that we do need this validation because of analyzers that may have changed and are outside of the mappings but part of index settings.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)