Enforcing CloudWatch log group retention period throughout the AWS Organization

[robertcurcio]

Is your feature request related to a problem? Please describe.

We would like to enforce the CloudWatch Log retention period throughout the AWS Organization. We have a lot of older CloudWatch log groups that have to be manually modified when we change our global retention settings.

• Each CloudWatch log group(s) need their retention policies updated after the modification of the Global-config.yaml cloudwatchLogRetentionInDays has modified. Only newly created CloudWatch log groups receive the new log retention policy.

I can see the AWS Lambda function in every account "AWSAccelerator-LoggingSta-NewCloudWatchLogsCreate" adds the our global retention policy onto newly created CloudWatch log groups. This function is triggered by the event bridge rule "CreateLogGroup" from CloudTrail. Unfortunately, it does not modify existing log groups.

Describe the feature you'd like

• A new event bus rule which triggers AWSAccelerator-LoggingSta-NewCloudWatchLogsCreate when it sees PutRetentionPolicy . This allows log group retention policy modifications by an AWS engineer to be reverted back to the global config.

• A scheduled job running periodically triggering function "AWSAccelerator-LoggingSta-NewCloudWatchLogsCreate" to find and remediates modified out of scope retention policies, and revers them back to the Global configuration

[richardkeit]

Hello @robertcurcio ,

For restricting modification of LogGroups, suggest that a SCP is created to restrict modification. Note: Should allow the principals that the solution runs as (AND if you want your own internal automation roles to have this ability).

Correct, there is a custom resource that modifies new log groups, there is also another custom resource which modifies existing log groups.

landing-zone-accelerator-on-aws/source/packages/@aws-accelerator/accelerator/lib/stacks/logging-stack.ts

Lines 1099 to 1110 in 26e8805

	// Run a custom resource to update subscription, KMS and retention for all existing log groups
	const customResourceExistingLogs = new CloudWatchLogsSubscriptionFilter(this, 'LogsSubscriptionFilter', {
	logDestinationArn: logsDestinationArnValue,
	logsKmsKey: this.cloudwatchKey,
	logArchiveAccountId: this.props.accountsConfig.getLogArchiveAccountId(),
	logsRetentionInDays: this.props.globalConfig.cloudwatchLogRetentionInDays.toString(),
	subscriptionFilterRoleArn: subscriptionFilterRole.roleArn,
	logExclusionOption: accountRegionExclusion,
	replaceLogDestinationArn: this.props.globalConfig.logging.cloudwatchLogs?.replaceLogDestinationArn,
	acceleratorPrefix: this.props.prefixes.accelerator,
	useExistingRoles: this.props.useExistingRoles ?? false,
	});

It is important to note, for log groups: the code will not update the log groups if the existing retention is longer than the default retention - ie in summary, the higher retention wins

landing-zone-accelerator-on-aws/source/packages/@aws-accelerator/constructs/lib/aws-cloudwatch-logs/update-subscription-filter/index.ts

Lines 161 to 177 in 4bb52f6

	async function updateRetentionPolicy(acceleratorRetentionInDays: number, logGroup: LogGroup) {
	const currentRetentionInDays = logGroup.retentionInDays;
	if (!currentRetentionInDays) {
	return;
	}
	if (acceleratorRetentionInDays > currentRetentionInDays) {
	await throttlingBackOff(() =>
	logsClient.send(
	new PutRetentionPolicyCommand({
	logGroupName: logGroup.logGroupName!,
	retentionInDays: acceleratorRetentionInDays,
	}),
	),
	);
	}
	}