Aggregating _ignored field values

[aclowkey]

When dealing with lots of data and lots of users, having the ignore_malformed option is great!
And using it in combination with _ignored field can give a lot of information about what is wrong with the data.

But unfortunately we can't aggregate on it, so it's hard to give an overview of which fields had issues in the last X hours.

I think it would be very useful to add .keyword for _ignored field, or make the _ignored field tokenised.

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

[jtibshirani]

We had an initial discussion and agreed that it seems useful to be able to aggregate over the _ignored field. We didn't reach a final conclusion but plan to continue the discussion. Other notes:

• We'd like to avoid adding another option to the mapping like "doc_values": true. However we weren't sure if it was worth the cost to always enable doc_values.
• One idea was to always enable doc_values, but stop adding the field as a stored field. The fetch phase would switch to retrieve _ignored from doc values.

[aclowkey]

Thanks for responding! What's the cost difference for enabling doc_values? I mean at scale there could be lots of _ignored fields, could the cost be amplified hugely?

…

On Mon, 3 Aug 2020 at 23:50, Julie Tibshirani ***@***.***> wrote: We had an initial discussion and agreed that it seems useful to be able to aggregate over the _ignored field. We didn't reach a final conclusion but plan to continue the discussion. Other notes: - We'd like to avoid adding another option to the mapping like "doc_values": true. However we weren't sure if it was worth the cost to always enable doc_values. - One idea was to always enable doc_values, but stop adding the field as a stored field. The fetch phase would switch to retrieve _ignored from doc values. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#59946 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPXFWN4F6HGRMMLISJO7XDR64PJNANCNFSM4PDHS2KQ> .

[davishmcclurg]

We recently started looking at using ignore_malformed in Enterprise Search. Being able to aggregate on the _ignored field would be super helpful for showing users what field types make sense for their data. Let me know if you're interested and I can explain a little bit more about our use case.

[jimczi]

We discussed internally and agreed to always enable doc_values, but stop adding the field as a stored field. The fetch phase would switch to retrieve _ignored from doc values as Julie proposed. @aclowkey would you still be interested to address this in your PR ?

[aclowkey]

@jimczi I addressed it. I hope I didn't misunderstand you.
Please review

[javanna]

This came up again as we are using the _ignored field also for fields that have a script configured. In case the script fails, the document may still get indexed (with on_script_error set to continue) but the field name goes in the _ignored field. Yet another reason to enable doc_values for the _ignored field.

[cbuescher]

I came across this issue while looking at the '_ignored' field mapper for #78981 I came across this issue and was wondering if we still want to pursue it.
Talking to @romseygeek he suggested that we might be able to use runtime fields for aggregation, but currently we don't seem to have access to
_ignored there either.

The following ideas came up during chatting:

• we might be able to access the stored fields from scripts, even if currently this might not work for runtime fields context, but other script contexts might already have this
• we can maybe add a special accessor to get _ignored
• we could as planned and enable doc_values, but stop adding the field as a stored field

If we find a solution that uses the existing way that '_ignored' is stored using "stored fields" and aggregate using runtime fields, this would have the additional benefit of working with existing indexes.

[cbuescher]

but currently we don't seem to have access to _ignored there either.

I played around a bit more with runtime field scripts and think I missed an option. Using:

POST /test/_search
{
  "query": {
    "exists": {
      "field": "_ignored"
    }
  },
  "runtime_mappings": {
      "my_ignored": {
        "type": "keyword",
        "script" : { "source" : """emit(params['_fields']._ignored.value)""" } 
      }
    },
  "aggs": {
    "a1": {
      "terms": {
        "field": "my_ignored"
      }
    }
  }
}

I now was able to aggregate over at least one value from the ' _ignored' field. I didn't get multiple values working (which would be necessary for an aggregation I think), but that might also already be possible with the right scripting magic.

[nik9000]

    "script" : { "source" : """emit(params['_fields']._ignored.value)""" }

I think

"""
for (def v : params['_fields']._ignored.values) {
  emit(v);
}
"""

We'd talked about a version of emit that took a list but never built it. It'd more complex than it sounds like.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)