Skip to content

Commit

Permalink
Group S3 options by source
Browse files Browse the repository at this point in the history
I am trying to group the S3 options by source:

- sqs queue
- bucket
  - aws bucket arn
  - non-aws bucket name

With this approach, we can define shared options like
`number_of_workers` only once. This should streamline the options and
avoid duplicated definitions.
  • Loading branch information
zmoog committed Aug 9, 2023
1 parent ba993cb commit 77d4e0f
Showing 1 changed file with 26 additions and 29 deletions.
55 changes: 26 additions & 29 deletions packages/aws_logs/data_stream/generic/agent/stream/aws-s3.yml.hbs
Original file line number Diff line number Diff line change
Expand Up @@ -4,57 +4,54 @@ data_stream:
pipeline: {{pipeline}}
{{/if}}

{{! The aws-s3 input can be configured to read from an SQS queue or an S3 bucket. }}

{{! start SQS queue }}
{{#unless bucket_arn}}
{{#unless non_aws_bucket_name}}
{{#if queue_url }}
queue_url: {{ queue_url }}
{{/if}}
{{/unless}}
{{/unless}}
{{! end SQS queue }}

{{#unless queue_url}}
{{#unless non_aws_bucket_name}}
{{#if bucket_arn }}
bucket_arn: {{ bucket_arn }}
{{/if}}
{{#unless queue_url}}{{! start S3 bucket }}

{{!
When using an S3 bucket, you can specify only one of the following options:
- An AWS bucket ARN
- A non-AWS bucket name
}}

{{! shared S3 options }}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}

{{#if bucket_list_prefix }}
bucket_list_prefix: {{ bucket_list_prefix }}
{{/if}}

{{#if bucket_list_interval }}
bucket_list_interval: {{ bucket_list_interval }}
{{/if}}
{{/unless}}

{{! AWS S3 bucket ARN options }}
{{#unless non_aws_bucket_name}}
{{#if bucket_arn }}
bucket_arn: {{ bucket_arn }}
{{/if}}
{{/unless}}

{{! non-AWS S3 bucket ARN options }}
{{#unless bucket_arn}}
{{#unless queue_url}}
{{#if non_aws_bucket_name }}
non_aws_bucket_name: {{ non_aws_bucket_name }}
{{/if}}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}
{{#if bucket_list_interval }}
bucket_list_interval: {{ bucket_list_interval }}
{{/if}}
{{/unless}}
{{/unless}}

{{#unless queue_url}}
{{#unless non_aws_bucket_name}}
{{#unless bucket_arn }}
{{#if bucket_list_prefix }}
bucket_list_prefix: {{ bucket_list_prefix }}
{{/if}}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}
{{#if bucket_list_interval }}
bucket_list_interval: {{ bucket_list_interval }}
{{/if}}
{{/unless}}
{{/unless}}
{{/unless}}
{{/unless}}{{! end S3 bucket }}

{{#if buffer_size }}
buffer_size: {{ buffer_size }}
Expand Down

0 comments on commit 77d4e0f

Please sign in to comment.