Generated artifacts for composable index templates

[ebeahan]

Summary

Include support for index template v2 in ECS by generating component templates and using those components to compose a v2 template. Users may also find value in adopting ECS published component templates into their own composable templates.

Background

Elasticsearch has introduced the second version of index templates, known as composable templates. A composable template is made up of component templates. Component templates are re-usable configuration building blocks that configure mappings, settings, and aliases. More details around can be found in the Elasticsearch docs and elastic/elasticsearch#53101.

[webmat]

I wonder if folks have feedback on what level of composability would be most helpful.

Currently we provide a sample Elasticsearch template that includes all the fields. Folks can adjust it using the scripts in this repo, following USAGE.md to add custom fields, remove unneeded fields for a given index/data source.

A few different approaches could be taken, in terms of how granular we make these new component templates:

• One component template for core fields, one for extended fields
• One component template per field set
  • Including fields of both levels in the same component template
  • Or one per field set & level (e.g. event-core & event-extended)
• More holistic component templates per use case (e.g. one for web logs, one for firewalls, one for authentications...)

The first two are the low hanging fruits. I think one component template per field set may be the most straightforward one to implement, and easiest to understand for users.

[jaakoob]

I would be most interested in component templates per field set which in my opinion should include fields from all levels.
How would you approach more holistic component templates? Adding additional tags to each element which mark the use case?

[webmat]

Indeed, more holistic templates would require more work to put together in ECS, and we'd need to e.g. tag each field set or event parts of field sets according to which use cases it's generally used in. Although I get the sense that this approach would be difficult to get right.

[jaakoob]

Yes, it seems that there would always be some case which would be forgotten with tagging.

[a03nikki]

I would like to see a component template per top level object in the ECS. For example, something like this

PUT _component_template/ecs-mapping-base
{
  "template": {
    "mappings": {
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": false,
      "_source": {
        "enabled": true,
        "includes": [],
        "excludes": []
      },
      "_meta": {
        "version": "1.7.0"
      },
      "_routing": {
        "required": false
      },
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "labels": {
          "type": "object"
        },
        "message": {
          "type": "text",
          "norms": false
        },
        "tags": {
          "type": "keyword",
          "ignore_above": 1024
        }
      }
    }
  },
  "_meta": {
    "original": "https://github.com/elastic/ecs/blob/1.7/generated/elasticsearch/7/template.json",
    "documentation": "https://www.elastic.co/guide/en/ecs/current/ecs-base.html"
  }
}

PUT _component_template/ecs-mapping-event
{
  "template": {
    "mappings": {
      "dynamic_templates": [],
      "properties": {
        "event": {
          "type": "object",
          "properties": {
            "action": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "category": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "created": {
              "type": "date"
            },
            "dataset": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "duration": {
              "type": "long"
            },
            "end": {
              "type": "date"
            },
            "hash": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "ingested": {
              "type": "date"
            },
            "kind": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "module": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "original": {
              "type": "keyword",
              "doc_values": false,
              "ignore_above": 1024,
              "index": false
            },
            "outcome": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "provider": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "reason": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "reference": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "risk_score": {
              "type": "float"
            },
            "risk_score_norm": {
              "type": "float"
            },
            "sequence": {
              "type": "long"
            },
            "severity": {
              "type": "long"
            },
            "start": {
              "type": "date"
            },
            "timezone": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "url": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        }
      }
    }
  },
  "_meta": {
    "original": "https://github.com/elastic/ecs/blob/1.7/generated/elasticsearch/7/template.json",
    "documentation": "https://www.elastic.co/guide/en/ecs/current/ecs-event.html"
  }
}

Then I could take these top level objects to create the _index_template for a (custom) given source. For example, a webserver access log would need the ecs-mapping-base, ecs-mapping-event, ecs-mapping-http, etc component templates. The top level objects could be added to the index template based on what fields were in the log file. The next custom source, would use a different sub-set of the ECS component templates.

When I'm doing custom source mappings, I don't pay attention to if a field's level is core or extended. I'm just looking at where does this piece of data best fit to achieve (1) standard field names for memorization, (2) multi log source searching, and (3) best use of the prebuilt content for Security and Observability.

[webmat]

Thanks for chiming in @a03nikki! I like these meta doc links too, that's a great idea ❤️

The plan for now is indeed to go with component templates per field sets (or top level objects).

It's on our plate to do in the near term. And since these aren't schema changes, we could backport this to the 1.7 branch, even if 1.7.0 is already released. The way we'll do it is these component templates will be new artifacts. The current "monolithic" template will remain as it is.

Do you think it would be helpful to backport even further, like 1.6? How common is it for you and colleagues to work on projects that are one or two ECS versions behind? Or do you simply always use the most recent version?