Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws-events-targets: KinesisFirehoseStream not accepting IDeliveryStream for imported DeliveryStream #25451

Closed
pags opened this issue May 5, 2023 · 6 comments · Fixed by #30189
Labels
@aws-cdk/aws-events-targets bug This issue is a bug. effort/small Small work item – less than a day of effort p1

Comments

@pags
Copy link

pags commented May 5, 2023

Describe the bug

When trying to add a KinesisFirehoseStream target to an event Rule, an error is thrown out of the aws-iam module.

Expected Behavior

CDK will successfully create an event rule with a target of an existing Kinesis Firehose stream.

Current Behavior

An error is thrown:

util_1.sum(values.map(v => (cdk.Token.isUnresolved(v) ? tokenSize : v.length) + 3 /* quotes, separator */));

TypeError: Cannot read properties of undefined (reading 'length')
    at @aws-cdk/aws-iam/lib/policy-statement.js:551:91

Reproduction Steps

new events.Rule(this, 'rule', {
			eventPattern: {
				source: ['aws.s3'],
				detail: {
					eventName: ['PutObject'],
					requestParameters: {
						bucketName: [bucket.bucketName]
					}
				}
			}
		}).addTarget(new targets.KinesisFirehoseStream(firehose.DeliveryStream.fromDeliveryStreamArn(this, 'firehose', 'my arn here'))); 

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.77.0 (build 06a0b19)

Framework Version

No response

Node.js Version

v16.14.2

OS

Ubuntu

Language

Typescript

Language Version

No response

Other information

No response

@pags pags added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels May 5, 2023
@github-actions github-actions bot added the @aws-cdk/aws-iam Related to AWS Identity and Access Management label May 5, 2023
@pahud pahud self-assigned this May 5, 2023
@pahud
Copy link
Contributor

pahud commented May 5, 2023

I believe this is because targets.KinesisFirehoseStream() is accepting CfnDeliveryStream as its parameter but fromDeliveryStreamArn actually returns IDeliveryStream instead.

image

I guess we probably need to allow KinesisFirehoseStream to accept the interface instead.

constructor(private readonly stream: firehose.CfnDeliveryStream, private readonly props: KinesisFirehoseStreamProps = {}) {

@pahud pahud added p1 effort/small Small work item – less than a day of effort and removed needs-triage This issue or PR still needs to be triaged. labels May 5, 2023
@pahud pahud removed their assignment May 5, 2023
@pahud pahud added @aws-cdk/aws-kinesisfirehose Related to Amazon Kinesis Data Firehose @aws-cdk/aws-events-targets and removed @aws-cdk/aws-iam Related to AWS Identity and Access Management labels May 5, 2023
@pahud pahud changed the title events: IAM Policy Error aws-events-targets: KinesisFirehoseStream not accepting IDeliveryStream for imported DeliveryStream May 5, 2023
@pags
Copy link
Author

pags commented May 5, 2023

Thank you for the insight, I am using JS (not TS) so did not have any type hints. Is there a prescribed way to get a CfnDeliveryStream instance based on what I have now?

@peterwoodworth
Copy link
Contributor

Assuming you've created a deliverystream like so:

const stream = new firehose.DeliveryStream(this, 'Delivery Stream', {
  ...
});

You can access the CfnDeliveryStream like so with escape hatches

const cfnStream = stream.node.defaultChild

@peterwoodworth
Copy link
Contributor

Sorry, you're importing it. If your DeliveryStream isn't managed by CloudFormation or CDK at all, you could cdk import the resource to be managed by your stack, and then the above comment could work. Else, if it's managed by CDK or another CloudFormation stack already and you want to keep it within that stack, I'm not sure there's a workaround here

@peterwoodworth peterwoodworth removed the @aws-cdk/aws-kinesisfirehose Related to Amazon Kinesis Data Firehose label May 5, 2023
@mergify mergify bot closed this as completed in #30189 May 16, 2024
mergify bot pushed a commit that referenced this issue May 16, 2024
…eam for imported deliverystream (#30189)

### Issue # (if applicable)

Closes #25451

### Reason for this change

Current events targets implementation only support L1 Delivery Stream as the input. We should support L2 IDeliveryStream as well for imported kinesis firehose stream.

### Description of changes

Add a V2 class to support kinesis firehose stream.

### Description of how you validated changes

New tests and existing tests pass.

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

atanaspam pushed a commit to atanaspam/aws-cdk that referenced this issue Jun 3, 2024
…eam for imported deliverystream (aws#30189)

### Issue # (if applicable)

Closes aws#25451

### Reason for this change

Current events targets implementation only support L1 Delivery Stream as the input. We should support L2 IDeliveryStream as well for imported kinesis firehose stream.

### Description of changes

Add a V2 class to support kinesis firehose stream.

### Description of how you validated changes

New tests and existing tests pass.

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@Reiss-Cashmore
Copy link

Reiss-Cashmore commented Aug 5, 2024

Okay so I'm really struggling with a mismatch here:

I'm creating a DeliveryStream


import { CfnDeliveryStream } from 'aws-cdk-lib/aws-kinesisfirehose';


const firehoseStream = new CfnDeliveryStream(
      this,
      generateName('subscription-fulfillment-firehose-delivery-stream'),
      {
        deliveryStreamType: 'DirectPut',
        extendedS3DestinationConfiguration: {
          bucketArn: firehoseBucket.bucketArn,
          roleArn: firehoseRole.roleArn,
          bufferingHints: {
            sizeInMBs: 64,
            intervalInSeconds: 300,
          },
          s3BackupMode: 'Enabled',
          s3BackupConfiguration: {
            bucketArn: firehoseBucket.bucketArn,
            roleArn: firehoseRole.roleArn,
            compressionFormat: 'GZIP',
          },
          dataFormatConversionConfiguration: {
            enabled: true,
            inputFormatConfiguration: {
              deserializer: {
                openXJsonSerDe: {},
              },
            },
            outputFormatConfiguration: {
              serializer: {
                parquetSerDe: {},
              },
            },
            schemaConfiguration: {
              databaseName: glueDatabase.databaseName,
              tableName: glueTable.tableName,
              roleArn: firehoseRole.roleArn,
              region: this.region,
              versionId: 'LATEST',
            },
          },
          compressionFormat: 'UNCOMPRESSED',
          prefix: 'event/',       
            errorOutputPrefix: 'error/',
          dynamicPartitioningConfiguration: {
            enabled: true,
            retryOptions: {
              durationInSeconds: 300,
            },
          },
          processingConfiguration: {
            enabled: true,
            processors: [
              {
                type: 'MetadataExtraction',
                parameters: [
                ],
              },
            ],
          },
        },
      },
    );

Then I create a rule and try to add the DeliveryStream as a target. No matter what I try here I can't seem to resolve this. I can't use the L2 DeliveryStream construct because I need access to a lot of the configuration parameters as above.

import {
  Rule
} from 'aws-cdk-lib/aws-events';

 // Firehose specific rule
              const firehoseRule = new Rule(
                this,
                generateName(
                  `eventbus-firehose-rule-${Id}-${subIdx}`,
                ),
                {
                  eventBus,
                  eventPattern: {
                    source: ['test'],
                    detailType: ['test'],
                    detail: fullPattern,
                  },
                },
              );

              firehoseRule.addTarget(
                new targets.KinesisFirehoseStreamV2(firehoseStream, {
                  deadLetterQueue: subscriptionDLQ,
                  event: RuleTargetInput.fromObject({
                    inputPathsMap: {
                      -snip-
                    },
                    inputTemplate: `{
                      -snip-
                    }`
                  }),
                }),
              );
            },
          );

In your changes @pahud @atanaspam @GavinZZ did you mean to stop supporting CfnDeliveryStream as a type for this method. If so, how do I reconcile a CfnDeliveryStream to match an IDeliveryStream?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-events-targets bug This issue is a bug. effort/small Small work item – less than a day of effort p1
Projects
None yet
4 participants