StackOverflowError when parsing deeply nested mappings

[ppf2]

Have a scenario where ES is throwing a StackOverflowError when the data node receives a cluster state published by the master. It is failing when parsing the mappings.

2015-10-27 00:23:50,561 [DEBUG] [service] [node_name] set local cluster state to version 575475 
2015-10-27 00:23:50,646 [WARN] [cluster] [node_name] [index_name] failed to add mapping [log], source [{"log":{"dynamic_templates":[{"facet_string_template":{"mapping":{"type":"multi_field","fields": 
... 
<a TON of mappings> 
... 
java.lang.StackOverflowError 
at org.elasticsearch.common.Strings.toUnderscoreCase(Strings.java:959) 
at org.elasticsearch.index.mapper.core.TypeParsers.parseField(TypeParsers.java:164) 
at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:153) 
at org.elasticsearch.index.mapper.core.TypeParsers.parseMultiField(TypeParsers.java:284) 
at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:184) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:189) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:189) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:189) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:189) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:189) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:189) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:189) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:189) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:189) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:290) 
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:214) 

In this particular case, there is an explosion in mapped fields due to dynamically created fields where a single field ends up having many many levels of nesting (recursion). As a result, StackOverflowError shows up whenever the mappings has to be parsed so that the nodes cannot process the updates.

It will probably be helpful if we have a sensible default to limit the number of depth levels for a field, etc..

[dakrone]

@ppf2 it sounds reasonable to me to have a sensible default to the limit. Do you know how deep these mappings went? Could we reasonably pick "100" as the limit for nesting?

[ppf2]

@dakrone Will send you a sample offline. I think we can choose a default, and also expose this as a configurable setting?

[rjernst]

100 seems crazy high to me. foo.bar.baz.a.b.c.d.e.f.g is already 10 levels deep and that is crazy...

[dakrone]

@ppf2 We don't need a sample, I was just curious the number of levels it went down (not the actual mapping)

[ppf2]

@dakrone In this case, easily > 500 levels 😮

[clintongormley]

Closing in favour of #11511