Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Add <type>@custom component template to each integration index template's composed_of array #190730

Closed
2 tasks done
kpollich opened this issue Aug 19, 2024 · 5 comments
Closed
2 tasks done

[Fleet] Add <type>@custom component template to each integration index template's composed_of array #190730

kpollich opened this issue Aug 19, 2024 · 5 comments
Assignees
@jillguyonnet
Labels
Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@kpollich
Copy link
Member

kpollich commented Aug 19, 2024 •
edited by jillguyonnet
Loading

To better support user customization across all documents of a given type, Fleet should add a @custom component template to each integration data stream's respective index template. For example, the index template for the logs-system.auth-default data stream should include logs@custom in its composed_of array:

logs@settings
logs-system.auth@package
logs@custom <---- Global customizations to all documents of type `logs` go here
logs-system.auth@custom
ecs@mappings
.fleet_globals-1
.fleet_agent_id_verification-1

Implementation checklist

  • @custom component templates are included for all types: logs, metrics, traces, synthetics, and profiling
  • Documentation is added to Fleet's data streams docs regarding the new component templates
@kpollich kpollich added the Team:Fleet Team label for Observability Data Collection Fleet team label Aug 19, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@felixbarny
Copy link
Member

A note on the ordering: in order for the <type>@custom template to be able to override field mappings from the package definition, it should go after @package, but before the <type>-<package>@custom template that's more specific to the package. For example:

logs@settings
logs-system.auth@package
logs@custom <---- Global customizations to all documents of type `logs` go here
logs-system.auth@custom
ecs@mappings
.fleet_globals-1
.fleet_agent_id_verification-1

@mbudge
Copy link

mbudge commented Aug 28, 2024

A note on the ordering: in order for the <type>@custom template to be able to override field mappings from the package definition, it should go after @package, but before the <type>-<package>@custom template that's more specific to the package. For example:

logs@settings
logs-system.auth@package
logs@custom <---- Global customizations to all documents of type `logs` go here
logs-system.auth@custom
ecs@mappings
.fleet_globals-1
.fleet_agent_id_verification-1

Are you sure?

Doesn't it need to go above "logs-system.auth@package" to override the package settings?

I've been wanting to add the lowercase normaliser to fields like host.name, user.name and user.target.name since we moved to Elastic-Agent. This is because users are constantly missing logs or struggling to search for hostnames where most search interfaces/KQL is case-sensitive. We want to add these lowercase normalisers globally to all log data, overriding the elastic integration mappings.

@felixbarny
Copy link
Member

Yes, later component templates have a higher precedence.

See also https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-template.html#multiple-component-templates

When multiple component templates are specified in the composed_of field for an index template, they are merged in the order specified, meaning that later component templates override earlier component templates.

This was referenced Sep 12, 2024
Merged
Merged
jillguyonnet added a commit that referenced this issue Sep 20, 2024
@jillguyonnet
[Fleet] Add <type>@Custom component template to integrations index te… (
c8ff7ea 
#192731)

## Summary

Relates #190730

This PR adds a `<type>@Custom `component template to integrations index
template's `composed_of` array.

### Testing

Integration install/update should be tested.

From my manual testing, Fleet seems to behave normally and the new
component template is added to the `composed_of` array. For example, the
output of `GET /_index_template/logs-system.auth` is:
```json
{
  "index_templates": [
    {
      "name": "logs-system.auth",
      "index_template": {
        "index_patterns": [
          "logs-system.auth-*"
        ],
        "template": {
          "settings": {},
          "mappings": {
            "_meta": {
              "package": {
                "name": "system"
              },
              "managed_by": "fleet",
              "managed": true
            }
          }
        },
        "composed_of": [
          "logs@mappings",
          "logs@settings",
          "logs-system.auth@package",
          "logs@custom",
          "logs-system.auth@custom",
          "ecs@mappings",
          ".fleet_globals-1",
          ".fleet_agent_id_verification-1"
        ],
        "priority": 200,
        "_meta": {
          "package": {
            "name": "system"
          },
          "managed_by": "fleet",
          "managed": true
        },
        "data_stream": {
          "hidden": false,
          "allow_custom_routing": false,
          "failure_store": false
        },
        "ignore_missing_component_templates": [
          "logs@custom",
          "logs-system.auth@custom"
        ]
      }
    }
  ]
}
```

### Screenshot

<img width="956" alt="Screenshot 2024-09-19 at 16 44 57"
src="https://github.com/user-attachments/assets/7e1d8e67-1d78-45a7-bc8d-8e509f1a2533">

<img width="956" alt="Screenshot 2024-09-19 at 17 04 45"
src="https://github.com/user-attachments/assets/8b6bbaf3-34b7-4169-8ca8-2bc6a38fe60d">

### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
@mergify mergify bot mentioned this issue Sep 20, 2024
Merged
@gregorywychowaniec-zt
Copy link

Hello,

I have a problem with the component template logs@custom and logs@mappings for which I have created a topic on Elastic Discuss. Does this issue address my problem?

@nimarezainia nimarezainia mentioned this issue Sep 30, 2024
Open
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 4, 2024
@jillguyonnet
[Fleet] Add <type>@Custom component template to integrations index te… (
4037d6b 
elastic#192731)

## Summary

Relates elastic#190730

This PR adds a `<type>@Custom `component template to integrations index
template's `composed_of` array.

### Testing

Integration install/update should be tested.

From my manual testing, Fleet seems to behave normally and the new
component template is added to the `composed_of` array. For example, the
output of `GET /_index_template/logs-system.auth` is:
```json
{
  "index_templates": [
    {
      "name": "logs-system.auth",
      "index_template": {
        "index_patterns": [
          "logs-system.auth-*"
        ],
        "template": {
          "settings": {},
          "mappings": {
            "_meta": {
              "package": {
                "name": "system"
              },
              "managed_by": "fleet",
              "managed": true
            }
          }
        },
        "composed_of": [
          "logs@mappings",
          "logs@settings",
          "logs-system.auth@package",
          "logs@custom",
          "logs-system.auth@custom",
          "ecs@mappings",
          ".fleet_globals-1",
          ".fleet_agent_id_verification-1"
        ],
        "priority": 200,
        "_meta": {
          "package": {
            "name": "system"
          },
          "managed_by": "fleet",
          "managed": true
        },
        "data_stream": {
          "hidden": false,
          "allow_custom_routing": false,
          "failure_store": false
        },
        "ignore_missing_component_templates": [
          "logs@custom",
          "logs-system.auth@custom"
        ]
      }
    }
  ]
}
```

### Screenshot

<img width="956" alt="Screenshot 2024-09-19 at 16 44 57"
src="https://github.com/user-attachments/assets/7e1d8e67-1d78-45a7-bc8d-8e509f1a2533">

<img width="956" alt="Screenshot 2024-09-19 at 17 04 45"
src="https://github.com/user-attachments/assets/8b6bbaf3-34b7-4169-8ca8-2bc6a38fe60d">

### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

(cherry picked from commit c8ff7ea)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 4, 2024
@jillguyonnet
[Fleet] Add <type>@Custom component template to integrations index te… (
9bb65d7 
elastic#192731)

## Summary

Relates elastic#190730

This PR adds a `<type>@Custom `component template to integrations index
template's `composed_of` array.

### Testing

Integration install/update should be tested.

From my manual testing, Fleet seems to behave normally and the new
component template is added to the `composed_of` array. For example, the
output of `GET /_index_template/logs-system.auth` is:
```json
{
  "index_templates": [
    {
      "name": "logs-system.auth",
      "index_template": {
        "index_patterns": [
          "logs-system.auth-*"
        ],
        "template": {
          "settings": {},
          "mappings": {
            "_meta": {
              "package": {
                "name": "system"
              },
              "managed_by": "fleet",
              "managed": true
            }
          }
        },
        "composed_of": [
          "logs@mappings",
          "logs@settings",
          "logs-system.auth@package",
          "logs@custom",
          "logs-system.auth@custom",
          "ecs@mappings",
          ".fleet_globals-1",
          ".fleet_agent_id_verification-1"
        ],
        "priority": 200,
        "_meta": {
          "package": {
            "name": "system"
          },
          "managed_by": "fleet",
          "managed": true
        },
        "data_stream": {
          "hidden": false,
          "allow_custom_routing": false,
          "failure_store": false
        },
        "ignore_missing_component_templates": [
          "logs@custom",
          "logs-system.auth@custom"
        ]
      }
    }
  ]
}
```

### Screenshot

<img width="956" alt="Screenshot 2024-09-19 at 16 44 57"
src="https://github.com/user-attachments/assets/7e1d8e67-1d78-45a7-bc8d-8e509f1a2533">

<img width="956" alt="Screenshot 2024-09-19 at 17 04 45"
src="https://github.com/user-attachments/assets/8b6bbaf3-34b7-4169-8ca8-2bc6a38fe60d">

### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

(cherry picked from commit c8ff7ea)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jillguyonnet jillguyonnet

Labels
Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@felixbarny @kpollich @mbudge @elasticmachine @jillguyonnet @gregorywychowaniec-zt