Increase max compilation rate for Filebeat modules tests

[ruflin]

Module tests started to fail with the following message:

Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting

The reason for this is (I think) that we started to use scripts more heavily in our pipelines so we hit this limit. For the tests we can easily increase the limit to solve this. In production this could still be an issue when all pipelines are loaded at the same time and we need to find a solution here.

In parallel we should also investigate if there are some non parameterised scripts which get recompiled every time.

• This also updates a generated Postgres file which was not updated properly.

[ruflin]

@jakelandis A few questions about the above:

• When are scripts compile? When the pipeline is loaded or when the first event comes in?
• Do grok patterns somehow also account as scripts? Asking because of the error I got below when trying to reproduce this locally:

2018-12-24T08:54:51.139+0100	DEBUG	[elasticsearch]	elasticsearch/client.go:537	Bulk item insert failed (i=0, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalStateException: pipeline with id [filebeat-7.0.0-logstash-slowlog-pipeline-plain] could not be loaded, caused by [ElasticsearchParseException[Error updating pipeline with id [filebeat-7.0.0-logstash-slowlog-pipeline-plain]]; nested: GeneralScriptException[Failed to compile inline script [logstash.slowlog.timestamp] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; GeneralScriptException[Failed to compile inline script [logstash.slowlog.timestamp] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; org.elasticsearch.common.breaker.CircuitBreakingException: [script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalStateException: pipeline with id [filebeat-7.0.0-logstash-slowlog-pipeline-plain] could not be loaded, caused by [ElasticsearchParseException[Error updating pipeline with id [filebeat-7.0.0-logstash-slowlog-pipeline-plain]]; nested: GeneralScriptException[Failed to compile inline script [logstash.slowlog.timestamp] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; GeneralScriptException[Failed to compile inline script [logstash.slowlog.timestamp] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; org.elasticsearch.common.breaker.CircuitBreakingException: [script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting]","caused_by":{"type":"illegal_state_exception","reason":"pipeline with id [filebeat-7.0.0-logstash-slowlog-pipeline-plain] could not be loaded, caused by [ElasticsearchParseException[Error updating pipeline with id [filebeat-7.0.0-logstash-slowlog-pipeline-plain]]; nested: GeneralScriptException[Failed to compile inline script [logstash.slowlog.timestamp] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; GeneralScriptException[Failed to compile inline script [logstash.slowlog.timestamp] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; org.elasticsearch.common.breaker.CircuitBreakingException: [script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting]"}},"header":{"processor_type":"unknown"}}

The pipeline for the above can be found here: https://github.com/elastic/beats/blob/master/filebeat/module/logstash/slowlog/ingest/pipeline-plain.json It does not contain an scripts but quite a few grok patterns.

[ruflin]

@jakelandis An interesting addition here. The docs state the limit is 15/min, but the error above says 75/5min which is similar but no the same ...

[ycombinator]

CI is failing with:

Too many dynamic script compilations within, max: [100/1m]

So looks like even 100/1m is not enough 😬😬😬

[ruflin]

I set it now to 1000/1m and at least locally it works. This is mainly to get test back to green. We should figure out about the why afterwards.

[kaiyan-sheng]

Thanks for updating postgresql-9.6-new-timestamp.log-expected.json!