[Ingest Manager] Composable templates error

[neptunian]

There was an error trying to install the endpoint package. Some datasets were changed in a new version, but a currently installed one should not be affected. Need to investigate the cause of this error.

@kuisathaverat feel free to provide any other details

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[kuisathaverat]

We have updated to latest Kibana and Elastic Agent snapshots and the issue is gone.

  "name": "kibana-kibana-84f6f75fd-5fdqj",
  "uuid": "630874ab-62dd-432a-ad58-286a44b53bc4",
  "version": {
    "number": "8.0.0",
    "build_hash": "a0dc26f0958227435f19a2a01290f3c0454c6c49",
    "build_number": 34802,
    "build_snapshot": true
  },

            "Labels": {
                "description": "Agent manages other beats based on configuration provided.",
                "license": "Elastic License",
                "org.label-schema.build-date": "2020-07-16T08:44:59Z",
                "org.label-schema.license": "Elastic License",
                "org.label-schema.name": "elastic-agent",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.url": "https://www.elastic.co/products/beats/elastic-agent",
                "org.label-schema.vcs-ref": "4dcbde39492bdc3843034bba8db811c68cb44b97",
                "org.label-schema.vcs-url": "github.com/elastic/beats/v7",
                "org.label-schema.vendor": "Elastic",
                "org.label-schema.version": "8.0.0",
                "org.opencontainers.image.created": "2020-05-04 00:00:00+01:00",
                "org.opencontainers.image.licenses": "GPL-2.0-only",
                "org.opencontainers.image.title": "CentOS Base Image",
                "org.opencontainers.image.vendor": "CentOS"
            }

[ph]

cc @jonathan-buttner Can you also take a look?

[neptunian]

This error is caused after elasticsearch tests that the index template created from the components and its own properties are valid. In this case the index template metrics-endpoint.telemetry probably had a merging conflict with the component template metrics-endpoint.telemetry-mappings and we ended up with an error response when trying to PUT the metrics-endpoint.telemetry index template.

I am not sure why the error went away after Agent and Kibana were updated but its possible when something changed between the Endpoint versions and the fact we are not removing component templates during an uninstall of a package could have resulted in some old incompatible component templates left in ES if ES was not fresh. I think it's safe to close this and expect not to see it after #72959 and after we stop making breaking changes to packages and expecting the user to have a fresh copy of ES.

Package authors should have integration tests to ensure that their index templates are valid with their component templates before publishing.