Latest release available on ECR Public:
public.ecr.aws/metaphor/awslb2cwlogs:arm64-latest
public.ecr.aws/metaphor/awslb2cwlogs:amd64-latest
Stream AWS Load Balancer Logs that are delivered to S3 into CloudWatch Logs for use with features like CloudWatch Logs Insights.
Logs are loaded from S3 as they are created using an S3 Event Notification, which can be optionally buffered via an SQS Queue. The logs are then parsed into JSON and shipped to Cloudwatch Logs.
cfn/example.yml
shows how to deploy the Lambda alongside Classic, Application or Network Load Balancers. Use this template as a starting point for your deployment.
To access the public ECR image directly in a Lambda you will either need to create a pull through cache for ECR Public, or use the public image URI as the FROM
directive in your own private dockerfile.
If you are using a pull through cache, use the following format to reference the image in your Lambda Function, assuming your ECR Public pull through cache was prefixed "ecr-public".
${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/ecr-public/metaphor/awslb2cwlogs:arm64-latest
The Lambda takes the following environment variables:
LOG_GROUP_NAME
: The name of the Log Group to ship toLOAD_BALANCER_TYPE
: The load balancer type. Must beclassic
,application
ornetwork
PLAINTEXT_LOGS
: If set to anything will ship the plaintext log line instead of parsing it to JSON
The Lambda requires access similar to the following IAM Policy:
{
"Version": "2012-10-17T00:00:00.000Z",
"Statement": [
{
"Effect": "Allow",
"Action": "logs:DescribeLogGroups",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:DescribeLogStreams",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:*:logs:*:REPLACE_WITH_AWS_ACCOUNT_ID:log-group:REPLACE_WITH_LOG_GROUP_NAME"
},
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:*:s3:::REPLACE_WITH_LOG_BUCKET_NAME/*"
}
]
}
The Lambda supports invocation via S3 events or SQS events. To use with SQS you will need a Queue, Queue Policy and Event Source similar to the following Cloudformation snippet:
LogDeliveryQueue:
Type: AWS::SQS::Queue
DeletionPolicy: Delete
UpdateReplacePolicy: Delete
Properties:
QueueName: !Sub ${AWS::StackName}-logs
VisibilityTimeout: 180
MessageRetentionPeriod: 345600
SQSQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref LogDeliveryQueue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
AWS: "*"
Action: sqs:SendMessage
Resource: "*"
Condition:
ArnLike:
aws:SourceArn: !Sub arn:${AWS::Partition}:s3:::${AWS::AccountId}-${AWS::StackName}
LoggingSQSEventSource:
Type: AWS::Lambda::EventSourceMapping
Properties:
BatchSize: 10
Enabled: true
EventSourceArn: !GetAtt LogDeliveryQueue.Arn
FunctionName: !Ref LoadBalancerLogsToCloudWatchLambda
ScalingConfig:
MaximumConcurrency: 2
FunctionResponseTypes:
- ReportBatchItemFailures
Maintained by Rupert Bryant-Greene of Metaphor
Originally derived from https://github.com/amazon-archives/cloudwatch-logs-centralize-logs