Bug with mustache iterators when searching with array input

[maggieghamry]

6.5 (bin/elasticsearch --version):

Description of the problem including expected versus actual behavior:
I have search templates that leverage array parameters. The examples below are simplified versions of a search template I am working on. The examples execute the GET _render/template endpoint with different inputs and then describe the outputs that returned.

Steps to reproduce:

Example #1 - The following search template fails to render:

GET _render/template 
{ 
"source": """ 
{ 
"query": { 
"bool": { 
"must": [ 
{{#ids}} { "terms": {"id.keyword": {{#toJson}}ids{{/toJson}} }} {{/ids}} 
] 
} 
} 
} 
""", 
"params": { 
"ids": ["2473272002", "1074568002"] 
} 
} 

The error returned is:

{ 
"error": { 
"root_cause": [ 
{ 
"type": "json_parse_exception", 
"reason": "Unexpected character ('{' (code 123)): was expecting comma to separate Array entries\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@3c379279; line: 5, column: 69]" 
} 
], 
"type": "json_parse_exception", 
"reason": "Unexpected character ('{' (code 123)): was expecting comma to separate Array entries\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@3c379279; line: 5, column: 69]" 
}, 
"status": 500 
} 

Example #2 - Another way to write the original template is to use mustache iterators (https://github.com/mustache/spec/blob/master/specs/sections.yml#L141-L169)

GET _render/template 
{ 
"source": """ 
{ 
"query": { 
"bool": { 
"must": [ 
{{#ids}} { "terms": {"id.keyword": {{.}} }} {{/ids}} 
] 
} 
} 
} 
""", 
"params": { 
"ids": ["2473272002", "1074568002"] 
} 
} 

Example #2 fails with the same error as Example #1.

Example #3 - Interestingly, if I add another query to the must clause the template validates with no errors.

GET _render/template 
{ 
"source": """ 
{ 
"query": { 
"bool": { 
"must": [ 
{{#ids}} { "terms": {"id.keyword": {{.}} }}, {{/ids}} 
{ 
"term": { 
"siteCode": "USAT" 
} 
} 
] 
} 
} 
} 
""", 
"params": { 
"ids": ["2473272002", "1074568002"] 
} 
} 

Example #3 returns:

{ 
"template_output" : { 
"query" : { 
"bool" : { 
"must" : [ 
{ 
"terms" : { 
"id.keyword" : 2473272002 
} 
}, 
{ 
"terms" : { 
"id.keyword" : 1074568002 
} 
}, 
{ 
"term" : { 
"siteCode" : "USAT" 
} 
} 
] 
} 
} 
} 
} 

4. Another way to render the template by only sightly altering it is to add an additional parameter. If I add a boolean parameter that controls the rendering on the array parameter then the template will validate with no errors.

GET _render/template 
{ 
"source": """ 
{ 
"query": { 
"bool": { 
"must": [ 
{{#idsPopulated}} { "terms": {"id.keyword": {{#toJson}}ids{{/toJson}} }} {{/idsPopulated}} 
] 
} 
} 
} 
""", 
"params": { 
"idsPopulated": true, 
"ids": ["2473272002", "1074568002"] 
} 
} 

Example #4 returns:

{ 
"template_output" : { 
"query" : { 
"bool" : { 
"must" : [ 
{ 
"terms" : { 
"id.keyword" : [ 
"2473272002", 
"1074568002" 
] 
} 
} 
] 
} 
} 
} 
} 

There appears to be a bug with how search templates handle array inputs. Examples #1 and #2 are valid mustache and should return results similar to example #4. Example #3 is identical to example #2 with an additional must query but returns the valid template output.

However, the output from example #3 and #4 are not the same. I would have expected example #3 to return:

{ 
"template_output" : { 
"query" : { 
"bool" : { 
"must" : [ 
{ 
"terms" : { 
"id.keyword" : [ 
"2473272002", 
"1074568002" 
] 
} 
}, 
{ 
"term" : { 
"siteCode" : "USAT" 
} 
} 
] 
} 
} 
} 
} 

[elasticmachine]

Pinging @elastic/es-core-infra

[astefan]

@maggieghamry I think all the examples you provided have to escape the double-quotes, since the mustache elements are not valid JSON elements.
Also, I don't think I understand what you are trying to do: you are using a terms query that does accept a list of elements but at the same time you seem to want to create a list of must statements, each with a terms filter in it?

In any case, this would be a solution without using an additional parameter like in your example 4:

{
  "source": "{\"query\":{\"bool\":{\"must\":[{ \"terms\": {\"id.keyword\":{{#toJson}}ids{{/toJson}} }}]}}}",
  "params": {
    "ids": [
      "2473272002",
      "1074568002"
    ]
  }
}

If you believe what I mentioned above is not the issue, do please provide more details about what you want to achieve.

[maggieghamry]

Thank you, @astefan . I was hoping to use one search template to drive two different searches. Because of this, the Ids need to be optional. So, the escaped template would look something like this, which returns the same error:

GET _render/template
{
  "source": "{\"query\":{\"bool\":{\"must\":[ {{#ids}} { \"terms\": {\"id.keyword\": {{#toJson}}ids{{/toJson}}  }} {{\/ids}} ]}}}",
  "params": {
    "ids": [
      "2473272002",
      "1074568002"
    ]
  }
}

Error returned:

"type": "json_parse_exception",
        "reason": "Unexpected character ('{' (code 123)): was expecting comma to separate Array entries\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@7d370e17; line: 1, column: 89]"

The objective is to use the search template for 2 types of searches. We have "policy" rules embedded in the template that are fairly complex and should be the same for both searches. First, we want to search by a list of identifiers. Secondly, a search where several fields can be used to search. Both should have the complex “policy” filters applied. We want to use one search template so we don’t have to maintain the complex policy rules in multiple places.

[robertBrnnn]

Hi @maggieghamry @astefan
I imagine you found a solution to this by now ;-) I faced a similar issue today in a search template with optional index boost array.

The way I found around this is to check if the first item is not empty.
So the escaped template would be:

{
  "source": "{\"query\":{\"bool\":{\"must\":[ {{#ids.0}} { \"terms\": {\"id.keyword\": {{#toJson}}ids{{/toJson}}  }} {{\/ids.0}} ]}}}",
  "params": {
    "ids": [
      "2473272002",
      "1074568002"
    ]
  }
}

The output of which is:

{
    "template_output": {
        "query": {
            "bool": {
                "must": [
                    {
                        "terms": {
                            "id.keyword": [
                                "2473272002",
                                "1074568002"
                            ]
                        }
                    }
                ]
            }
        }
    }
}

Similarly, if the ids parameter isn't provided i.e.

{
  "source": "{\"query\":{\"bool\":{\"must\":[ {{#ids.0}} { \"terms\": {\"id.keyword\": {{#toJson}}ids{{/toJson}}  }} {{\/ids.0}} ]}}}",
  "params": {
    
  }
}

The output is:

{
    "template_output": {
        "query": {
            "bool": {
                "must": []
            }
        }
    }
}

The only caveat is that if you leave an empty array in, it'll cause an index_out_of_bounds_exception i.e

"params":{
    "ids": []
}

As I said you probably found a solution to this, but good to have it documented here for others in the future.

It'd be useful if this could be included in the elasticsearch documentation somewhere; I spent hours trying to find a solution to this.

Cheers,
Rob

[ishi]

Hi @maggieghamry

Studying closelly mustache documentation I can see that this behavior is according to specification

When the value is a non-empty list, the text in the block will be displayed once for each item in the list

As a workaround I would suggest to modify the input parameters structure to something like this

"params":{
  "ids": {
    "values": [
      "2473272002",
      "1074568002"
    ] 
  }
}

and template accordingly

{ 
  "query": { 
    "bool": { 
      "must": [ 
        {{#ids}} { "terms": {"id.keyword": {{#toJson}}values{{/toJson}} }} {{/ids}} 
      ] 
    } 
  } 
} 

This is probably the best you can get with mustache.

Cheers