Use unlimited compilation rate limit for mustache templates in Ingest

[andrewkroh]

In #59267 the compilation rate limit for the script and processor_conditional contexts was increased to unlimited. However some of the Ingest Node processors depend on mustache templates that are also scripts that need to be compiled. And those templates have their own separate rate limit. So users of Ingest Node pipelines can still hit limits. So I'd like to propose that templates also get an unlimited compilation rate in Ingest Node.

Here's an example of the type of error users see when using Filebeat modules or packages with Fleet:

Bulk item insert failed (i=5, status=500): {\"type\":\"illegal_state_exception\",\"reason\":\"pipeline with id [logs-system.auth-0.9.0] could not be loaded, caused by [ElasticsearchParseException[Error updating pipeline with id [logs-system.auth-0.9.0]]; nested: GeneralScriptException[Failed to compile inline script [{{system.auth.ssh.dropped_ip}}] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.context.template.max_compilations_rate] setting];; GeneralScriptException[Failed to compile inline script [{{system.auth.ssh.dropped_ip}}] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.context.template.max_compilations_rate] setting];; org.elasticsearch.common.breaker.CircuitBreakingException: [script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.context.template.max_compilations_rate] setting]\"}",

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Scripting)

[jakelandis]

@jdconrad @stu-elastic - I believe that the fix here would be in TemplateScript, but that is used in quite a few places. Do you think that we should create a new ingest_template context (and use that in ingest node) and increase that context limit to unlimited ?

[stu-elastic]

We plan on moving all non-search contexts to unlimited by 7.11.

[stu-elastic]

#62899

[alexander-marquardt]

If the cache size is too small for the number of scripts that the client is running, then this can result in each script
being pushed out of the cache, and then recompiled shortly thereafter -- thereby causing this "too many script compilations". In this case, making the cache size bigger (so compiled scripts are not evicted from the cache) may be more beneficial than increasing the compilation limit.

I have seen setting the compilation limit to a large number crash a cluster if the real problem is that the script cache is too small. (related support case: 00599383)

[stu-elastic]

All the compiled scripts are held onto by the ingest pipeline. I'll verify when I get to #62899 that templates are as well, but I believe they are.

Upshot is cache is irrelevant for ingest pipelines, they are never recompiled, they stay with the pipeline for it's lifetime.

[alexander-marquardt]

Good to know, thanks!

[kesslerm]

It may be necessary to delete (optional?) and upload the ingest pipeline again if the above error occurs repeatedly for a given pipeline and does not react to changing the compilation settings of the cluster. This is likely due to the error already being stored in the pipeline.

[stu-elastic]

If a pipeline definition hits a compilation limit, the pipeline will remember the error and will not function. The solution is to update the limit and then delete and add back the pipeline.