[AWS] Add number_of_workers and latency to all CloudWatch Logs based integrations

[zmoog]

Motivation

Number of workers

When users set up the integration to collect logs from a single log group, the default value (1) for number_of_workers is adequate to fetch the log events at each iteration.

However, when users utilize log_group_name_prefix to collect log events from multiple log groups, according to elastic/beats#29695, the number_of_workers should be increased to reach the number of log groups matching the prefix.

Latency

The latency can be required on the busiest log groups to deal with potential latency.

Conclusion

All the CloudWatch Logs-based integration should have these options available to them.

Change description

Updates the configuration template and integration variables to make these Filebeat configuration options available to the integration settings UI.

It also added a new "Advanced options" section under the "Setup" section to give more details about these options. I plan to add leverage this and other new sections to explain how the integration works.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

I created four logs groups with plain text data:

• test-mbranca-groucho
• test-mbranca-gummo
• test-mbranca-harpo
• test-mbranca-zeppo

And I configured a CloudWatch Logs integration with the following settings:

• Log Group Name Prefix: test-mbranca
• Region: <REGION FOR THE LOG GROUPS>
• All other settings have been left with their default value

The only existing worker will pick up one log group at a time during each iteration.

Each log group will process data every four iterations.

Then I changed the following settings:

• Number of workers: 4 (previously was unset)

The four available workers will pick up one log group each during each iteration.

Related issues

• Relates Support running multiple log groups in cloudwatch input beats#29695

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-04-07T17:01:08.327+0000

• Duration: 47 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	190
Skipped	4
Total	194

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (15/15)	💚
Files	93.75% (15/16)	👎 -3.073
Classes	93.75% (15/16)	👎 -3.073
Methods	85.921% (238/277)	👎 -6.198
Lines	85.925% (7387/8597)	👎 -5.349
Conditionals	100.0% (0/0)	💚

[elasticmachine]

Package aws - 1.33.3 containing this change is available at https://epr.elastic.co/search?package=aws

[TheLastHarpCo]

I don't know what I am doing or who Harpo is