[Ingest Manager] Updating a package immediately after a rollover fails

[jonathan-buttner]

Updating a package immediately after performing a rollover on a data stream fails.

Steps to reproduce:

1. Boot up a fresh kibana and elasticsearch
2. Force install an older endpoint package: POST http://elastic:changeme@localhost:5601/api/ingest_manager/epm/packages/endpoint-0.16.0-dev.3?force=true
3. Ingest some data that matches a data stream of the endpoint package

https://github.com/elastic/kibana/blob/9d07a677b89a723db55a3f307bfb09c7ae189de5/x-pack/test/functional/es_archives/endpoint/pipeline/dns/data.json.gz

node scripts/es_archiver.js --es-url http://elastic:changeme@localhost:9200 --kibana-url http://elastic:changeme@localhost:5601 load x-pack/test/functional/es_archives/endpoint/pipeline/dns --use-create

The above archive will populate data in the logs-endpoint.events.network-default data stream.

4. Perform a manual rollover POST /logs-endpoint.events.network-default/_rollover
5. Observe that the rolled over backing index does not have the value fields associated with the data_stream object:

6. Navigate to the Ingest manager app to perform an upgrade or do

POST http://elastic:changeme@localhost:5601/api/ingest_manager/epm/packages/_bulk
{
  packages: ['endpoint']
}

7. Observe the failure:

This is thrown here:

kibana/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.ts

Line 407 in c456f64

throw new Error(`data_stream values are missing from the index template ${indexName}`);

8. Ingest more data into the data stream (can probably use the same archive from step 3).
9. Observe that the backing index has the value fields populated under data_stream

10. Navigate back to the ingest manager and you will likely see another error

Or after some time this:

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

Pinging @elastic/endpoint-management (Team:Endpoint Management)

[jonathan-buttner]

cc: @nnamdifrankie @kevinlog

[jonathan-buttner]

@ruflin I reached to the elasticsearch team to see if it is expected behavior that the value fields will not be populated on a backing index immediately after a rollover. When you first install a data stream, there is no backing index so you don't run into this problem. And as soon as data is ingested the backing index will get the correct value fields.

[jonathan-buttner]

@ruflin I reached to the elasticsearch team to see if it is expected behavior that the value fields will not be populated on a backing index immediately after a rollover. When you first install a data stream, there is no backing index so you don't run into this problem. And as soon as data is ingested the backing index will get the correct value fields.

Actually I chatted with @danhermann and this is the expected behavior as stated by the docs:

https://www.elastic.co/guide/en/elasticsearch/reference/7.x/keyword.html#constant-keyword-field-type

In case no value is provided in the mappings, the field will automatically configure itself based on the value contained in the first indexed document...

I'm not sure if we'd want to do this, but we could add the value field to each property when the ingest manager constructs the template. Since it should always be based on the contents of the package right?

Except the namespace field, that one we'd want the tool sending the data to fill the field 🤔

[neptunian]

@jonathan-buttner thanks for creating this issue.

the only reason the error is thrown is because EPM does not know the namespace, which is why we also cant populate during template creation time.

I haven't tried it yet but I think we can pass mappings to the rollover, so perhaps we can pass the constants' values so they always get carried over.

[muskangulati-qasource]

Bug Conversion

• No Test Case Created for this ticket as its Dev Validation Ticket

Thanks!