"[" key in an event crashes logstash

[sribharghava]

For all general issues, please provide the following details for fast resolution:

• Version: 7.4.0
• Operating System: Linux
• Config File (if you have sensitive info, please remove it): Just read it with any input plugin, I encountered this while reading with elastic search input plugin.
• Sample Data: {"[": null}
• Steps to Reproduce: Try creating an event object with the sample data pasted above.
  It fails with the following error,
  Error: Invalid FieldReference: [ Exception: Java::OrgLogstash::FieldReference::IllegalSyntaxException Stack: org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:283)

As we don't have control on the incoming data, should logstash fail when it fails to parse the incoming data? Although it may look fair from the logstash perspective, this crash resets the elastic input plugin which then starts reading from the start again. One bad record results in an infinite loop! The fix I believe should come from logstash instead of elastic input plugin as this potentially can affect multiple other plugins as well. At least there should be a way to ignore such data errors while creating events at the logstash level.

[yaauie]

Can you provide a backtrace? I believe we have resolved a few of these since 7.4 shipped, but I'd be glad to help chase this instance down. Many plugins/codecs will catch this kind of exception and produce a tagged event that contains a serialized form of the payload.

[sribharghava]

Error: Invalid FieldReference: [Exception: Java::OrgLogstash::FieldReference::IllegalSyntaxException Stack: org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:283) org.logstash.FieldReference.parse(FieldReference.java:184) org.logstash.FieldReference.parseToCache(FieldReference.java:175) org.logstash.FieldReference.from(FieldReference.java:107) org.logstash.FieldReference.lambda$from$0(FieldReference.java:98) java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1737) org.logstash.FieldReference.from(FieldReference.java:98) org.logstash.ConvertedMap.convertKey(ConvertedMap.java:102) org.logstash.ConvertedMap.access$000(ConvertedMap.java:24) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:64) org.logstash.Valuefier.lambda$initConverters$12(Valuefier.java:151) org.logstash.Valuefier.convert(Valuefier.java:74) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:64) org.logstash.Valuefier.lambda$initConverters$12(Valuefier.java:151) org.logstash.Valuefier.convert(Valuefier.java:74) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:64) org.logstash.Valuefier.lambda$initConverters$12(Valuefier.java:151) org.logstash.Valuefier.convert(Valuefier.java:74) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:35) org.logstash.ConvertedMap$1.visit(ConvertedMap.java:29) org.jruby.RubyHash.visitLimited(RubyHash.java:689) org.jruby.RubyHash.visitAll(RubyHash.java:674) org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:69) org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_initialize(JrubyEventExtLibrary.java:72) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$method$push_hit$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb:162) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$block$run$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb:144) org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146) org.jruby.runtime.BlockBody.yield(BlockBody.java:114) org.jruby.runtime.Block.yield(Block.java:170) org.jruby.RubyArray.each(RubyArray.java:1800) org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen) org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:555) org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb:144) usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_elasticsearch_minus_4_dot_0_dot_2.lib.logstash.inputs.elasticsearch.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.0.2/lib/logstash/inputs/elasticsearch.rb) org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91) org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90) org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:314) usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb) org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91) org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90) org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:306) org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) org.jruby.runtime.Block.call(Block.java:129) org.jruby.RubyProc.call(RubyProc.java:295) org.jruby.RubyProc.call(RubyProc.java:274) org.jruby.RubyProc.call(RubyProc.java:270) org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) java.base/java.lang.Thread.run(Thread.java:834)

[Zoom2016]

I also have this situation, looking forward to solving it.
My question

[reneschroeder0000]

Any updates on this? This seems to still be an issue with 7.12.1

[fahadmaq007]

I resolved it by stringifying the incoming json document in the respective plug-in (couchdb_changes.rb). Works fine.
@Zoom2016 @reneschroeder0000

[balusarakesh]

any updates on this? this bug is causing our pipelines to crash and we have no way to fix this

[abdelbarre]

any updates on this?

[yaauie]

This was resolved in 8.3.0 with #14044.

Even with the above fix in place, the Elasticsearch input plugin should certainly be more resilient to failure in general, so I have filed logstash-plugins/logstash-input-elasticsearch#182