Remove lightweight modules in packages

[kaiyan-sheng]

Background

Before Elastic Agent and packages existed we introduced lightweight modules in Metricbeat to simplify the development of new metricsets and packages. The goal was to move to pure config defined modules as we have in Filebeat. Over time we have been making more and more use of it, for example in the AWS package. This is great as it reduces the number of code changes we have to make.

With the introduction of elastic packages, it has become even more important to have packages purely based on configurations because that is what is delivered as part of the package. Unfortunately many modules use lightweight modules but the packages we build also depend on the lightweight modules to be there, meaning the code/config files have to be contributed to the beats repository to work. This makes it easier to keep packages and modules in sync but it also means we keep adding to the beats repository. This issue is caused by running the migration script from Metricbeat to integrations.

Actions

1. For the old migrated data streams (from Bests) like rds: we should move the configuration part from Beats into integrations.
2. For all data streams: use ingest pipeline instead of rename processor.
3. For all new packages, we should add configuration directly into integrations.

Impacted Integrations (wip)

• aws dynamodb Remove lightweight modules from aws dynamodb data stream #3597
• aws ebs Remove lightweight modules from aws ebs data stream #3598
• aws lambda Remove lightweight modules from aws lambda data stream #3599
• aws natgateway Remove lightweight modules from aws natgateway data stream #3600
• aws s3_daily_storage Remove lightweight modules from aws s3_daily_storage data stream #3601
• aws s3_request Remove lightweight modules from aws s3_request data stream #3602
• aws sns Remove lightweight modules from aws sns data stream #3603
• aws sqs Remove lightweight modules from aws sqs data stream #3604
• aws transitgateway Remove lightweight modules from aws transitgateway data stream #3605
• aws usage Remove lightweight modules from aws usage data stream #3606
• aws kinesis [AWS] migrate AWS Kinesis module from beats to integrations  #3822
• azure_metrics container_instance Remove lightweight modules from azure_metrics container_instance data stream #3592
• azure_metrics container_registry Remove lightweight modules from azure_metrics container_registry data stream #3594
• azure_metrics container_service Remove lightweight modules from azure_metrics container_service data stream #3595
• azure_metrics database_account Remove lightweight modules from azure_metrics database_account data stream #3596
• gcp.billing [GCP] Remove lightweight module in gcp.billing data stream #3720
• gcp.compute [GCP] Remove lightweight module in gcp.compute data stream #3721
• gcp.firestore [GCP] Remove lightweight module in gcp.firestore data stream #3722
• gcp.storage [GCP] Remove lightweight module in gcp.storage data stream #4262
• gcp.pubsub [GCP] Remove lightweight module in gcp.pubsub data stream #4263
• gcp.loadbalancing [GCP] Remove lightweight module in gcp.loadbalancing data stream #4264
• gcp.dataproc [GCP] Remove lightweight module in gcp.dataproc data stream #4265

[ruflin]

++ on moving to a fully package based approach. There is another benefit of this that the data collection part on the edge becomes less complex and needs less changes. This might make it easier that also newer version of Elastic Agent can ship data to older versions of Elasticsearch as the data format shipped stays the same. What matters is only the package that is installed.

On side effect of 2 is that we will use the rename processor a lot more (see https://github.com/elastic/beats/blob/main/x-pack/metricbeat/module/aws/rds/manifest.yml#L79 as an example). Lets make sure to involve the es-perf team ( @DJRickyB ) in this. I'm not too concerned on the metrics front about this change as my understanding is the number of events is not very high compared to logs and we do much heavier processing on the logs side but it is worth investigating. It might be useful to have a bulk rename processor that could also make it easier to write these renames.

[kaiyan-sheng]

One problem showed up is: before moving configuration from Beats to Integrations, EC2 metrics data stream generates events with field:

    "metricset": {
        "period": 300000,
        "name": "ec2"
    }

With moving the configuration into integrations, EC2 metrics data stream generates events with field:

    "metricset": {
        "period": 300000,
        "name": "cloudwatch"
    }

This is caused by metricsets field in stream.yml.hbs. It changed from ec2 to cloudwatch after moving the configuration to integrations: https://github.com/elastic/integrations/blob/main/packages/aws/data_stream/ec2_metrics/agent/stream/stream.yml.hbs#L1

Should we add a rename processor to rename metricset.name from cloudwatch to ec2 in this case?

[ruflin]

The rename processor is the low hanging fruit here. More fundamental, I wonder if we still need the metricset.name field in the first place. Are we using it for any queries? In general it should be replaced by queries on data_stream.dataset as this is much more efficient.

[kaiyan-sheng]

@ruflin Agreed. metricset.name is not used on our side (dashboards). Users should also use data_stream.dataset to identify which service these metrics are from.

[endorama]

Should we add a removal step in the ingest pipeline to remove metricset.name in integrations?

[kaiyan-sheng]

Should we add a removal step in the ingest pipeline to remove metricset.name in integrations?

@endorama Good question! This field does tell us which metricset is it using from Metricbeat. This way, user knows we are using the cloudwatch metricset in the background instead of the ec2 metricset for example. Not sure how useful this is but since this field exist across all integrations, maybe we can keep it for now?

[tommyers-elastic]

@kaiyan-sheng @endorama @zmoog i think everything is done here - shall we close this issue once and for all?

[kaiyan-sheng]

@tommyers-elastic I just went though our GCP package and found these data streams are added but not with lightweight module removed. I will move these into the issue description.

• GCP storage
• GCP pubsub
• GCP loadbalancing
• GCP dataproc

[ChrsMark]

@kaiyan-sheng shall we communicate this effort with the wider audience of integrations' teams? This would help to advertise the recommended way when it comes to the usage (or non usage :)) of leighweight modules.

We discovered this while implementing/reviewing at #4253 (comment).
cc: @gsantoro

Also I guess the long term plan would be to use input packages (when they are ready). Relevant content: elastic/package-spec#319 (comment) and elastic/package-spec#404

[ruflin]

++, lets make sure it is document in a guide around how to build integrations.

[kaiyan-sheng]

@ruflin +1! I also suggested working on documentation for how to build integrations (especially with lightweight module) for our team for 8.6.0 planning.

[kaiyan-sheng]

@tommyers-elastic I think this meta issue is ready to be closed now.