Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AWS] Remove duplicated number_of_workers settings from the custom logs integration #7319

Merged
merged 4 commits into from
Aug 21, 2023
Merged

[AWS] Remove duplicated number_of_workers settings from the custom logs integration #7319

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ba993cb
Remove the accidental extra number_of_workers
zmoog Aug 9, 2023
77d4e0f
Group S3 options by source
zmoog Aug 9, 2023
1662abc
Update packages/aws_logs/data_stream/generic/manifest.yml
zmoog Aug 21, 2023
19b47e0
Small updates to template comments
zmoog Aug 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions packages/aws_logs/data_stream/generic/agent/stream/aws-cloudwatch.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,6 @@ api_sleep: {{ api_sleep }}
{{#if latency }}
latency: {{ latency }}
{{/if}}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}

{{#if credential_profile_name}}
credential_profile_name: {{credential_profile_name}}
Expand Down
55 changes: 26 additions & 29 deletions packages/aws_logs/data_stream/generic/agent/stream/aws-s3.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,57 +4,54 @@ data_stream:
pipeline: {{pipeline}}
{{/if}}

{{! The aws-s3 input can be configured to read from an SQS queue or an S3 bucket. }}

{{! start SQS queue }}
{{#unless bucket_arn}}
{{#unless non_aws_bucket_name}}
{{#if queue_url }}
queue_url: {{ queue_url }}
{{/if}}
{{/unless}}
{{/unless}}
{{! end SQS queue }}

{{#unless queue_url}}
{{#unless non_aws_bucket_name}}
{{#if bucket_arn }}
bucket_arn: {{ bucket_arn }}
{{/if}}
{{#unless queue_url}}{{! start S3 bucket polling }}

{{!
When using an S3 bucket, you can specify only one of the following options:
- An AWS bucket ARN
- A non-AWS bucket name
}}

{{! shared S3 bucket polling options }}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}

{{#if bucket_list_prefix }}
bucket_list_prefix: {{ bucket_list_prefix }}
{{/if}}

{{#if bucket_list_interval }}
bucket_list_interval: {{ bucket_list_interval }}
{{/if}}
{{/unless}}

{{! AWS S3 bucket ARN options }}
{{#unless non_aws_bucket_name}}
{{#if bucket_arn }}
bucket_arn: {{ bucket_arn }}
{{/if}}
{{/unless}}

{{! non-AWS S3 bucket ARN options }}
{{#unless bucket_arn}}
{{#unless queue_url}}
{{#if non_aws_bucket_name }}
non_aws_bucket_name: {{ non_aws_bucket_name }}
{{/if}}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}
{{#if bucket_list_interval }}
bucket_list_interval: {{ bucket_list_interval }}
{{/if}}
{{/unless}}
{{/unless}}

{{#unless queue_url}}
{{#unless non_aws_bucket_name}}
{{#unless bucket_arn }}
{{#if bucket_list_prefix }}
bucket_list_prefix: {{ bucket_list_prefix }}
{{/if}}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}
{{#if bucket_list_interval }}
bucket_list_interval: {{ bucket_list_interval }}
{{/if}}
{{/unless}}
{{/unless}}
{{/unless}}
{{/unless}}{{! end S3 bucket polling }}

{{#if buffer_size }}
buffer_size: {{ buffer_size }}
Expand Down
8 changes: 1 addition & 7 deletions packages/aws_logs/data_stream/generic/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ streams:
default: 1
required: false
show_user: false
description: Number of workers that will process the log groups with the given log_group_name_prefix. Default value is 1.
description: The number of workers assigned to read from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`.
- name: log_streams
type: text
title: Log Streams
Expand Down Expand Up @@ -95,12 +95,6 @@ streams:
required: false
show_user: false
description: "The amount of time required for the logs to be available to CloudWatch Logs. Sample values, `1m` or `5m` — see Golang [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for more details. Latency translates the query's time range to consider the CloudWatch Logs latency. Example: `5m` means that the integration will query CloudWatch to search for logs available 5 minutes ago."
- name: number_of_workers
type: integer
title: Number of workers
required: false
show_user: false
description: The number of workers assigned to reading from log groups. Each worker will read log events from one of the log groups matching `log_group_name_prefix`. For example, if `log_group_name_prefix` matches five log groups, then `number_of_workers` should be set to `5`. The default value is `1`.
- name: tags
type: text
title: Tags
Expand Down