Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PR - Fargate OSS pattern logs enabled | FluentBit #137

Merged
merged 12 commits into from
Feb 5, 2024
Binary file added docs/patterns/images/logs-fargate-fluentbit.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -194,8 +194,10 @@ You should now see a new dashboard named `Java/JMX`, under `Observability Accele

## Viewing Logs

By default, we deploy a FluentBit daemon set in the cluster to collect worker logs for all namespaces. Logs are collected and exported to Amazon CloudWatch Logs, which enables you to centralize the logs from all of your systems, applications,
and AWS services that you use, in a single, highly scalable service.
Amazon EKS on Fargate offers a built-in log router based on Fluent Bit. This means that you don't explicitly run a Fluent Bit container as a sidecar, but Amazon runs it for you. All that you have to do is configure the log router. The configuration happens through a dedicated [`ConfigMap`](https://github.com/aws-observability/cdk-aws-observability-accelerator/blob/main/lib/common/resources/fluent-bit/fluent-bit-fargate-config.ytpl). Logs are collected and exported to Amazon CloudWatch Logs, which enables you to centralize the logs from all of your systems, applications,
and AWS services that you use, in a single, highly scalable service. By default, the logs are exported to us-east-1 region but you can modify the `ConfigMap` for your region of choice. At least one supported `OUTPUT` plugin has to be provided in the `ConfigMap` to enable logging. You can also modify the destination from cloudwatch to Cloudwatch (default), Amazon OpenSearch Service or Kinesis Data Firehose. Read more about [EKS Fargate logging](https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html).

![fargate-fluentbit](../images/logs-fargate-fluentbit.png)

## Teardown

Expand Down
42 changes: 42 additions & 0 deletions lib/common/resources/fluent-bit/fluent-bit-fargate-config.ytpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
kind: Namespace
apiVersion: v1
metadata:
name: aws-observability
labels:
aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
name: aws-logging
namespace: aws-observability
data:
flb_log_cw: "{{enableFlbProcessLogs}}" # Set to true to ship Fluent Bit process logs to CloudWatch.
filters.conf: |
[FILTER]
Name parser
Match *
Key_name log
Parser crio
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
Buffer_Size 0
Kube_Meta_Cache_TTL 300s
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match kube.*
region {{awsRegion}}
log_group_name {{logGroupName}}
log_stream_prefix {{log_stream_prefix}}
auto_create_group true
parsers.conf: |
[PARSER]
Name crio
Format Regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ export default class SingleNewEksAWSNativeFargateobservabilityConstruct {
// Define fargate cluster provider and pass the profile options
const fargateClusterProvider : blueprints.FargateClusterProvider = new blueprints.FargateClusterProvider({
fargateProfiles,
version: eks.KubernetesVersion.of("1.27")
version: eks.KubernetesVersion.of("1.28")
elamaran11 marked this conversation as resolved.
Show resolved Hide resolved
});

const certManagerAddOnProps : blueprints.CertManagerAddOnProps = {
Expand All @@ -50,7 +50,7 @@ export default class SingleNewEksAWSNativeFargateobservabilityConstruct {
};

const coreDnsAddOnProps : blueprints.CoreDnsAddOnProps = {
version:"v1.10.1-eksbuild.1",
version:"v1.10.1-eksbuild.6",
configurationValues:{
computeType: "Fargate"
}
Expand All @@ -64,7 +64,6 @@ export default class SingleNewEksAWSNativeFargateobservabilityConstruct {
.withCertManagerProps(certManagerAddOnProps)
.withCoreDnsProps(coreDnsAddOnProps)
.enableFargatePatternAddOns()
.enableControlPlaneLogging()
.clusterProvider(fargateClusterProvider)
.addOns(...addOns)
.build(scope, stackId);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
import 'source-map-support/register';
import * as blueprints from '@aws-quickstart/eks-blueprints';
import { KubectlProvider, ManifestDeployment } from "@aws-quickstart/eks-blueprints/dist/addons/helm-addon/kubectl-provider";
import { loadYaml, readYamlDocument } from '@aws-quickstart/eks-blueprints/dist/utils';

/**
* Configuration options for the fluentbit configmap
*/
export interface FluentBitConfigMapProps {

/**
* Region to send cloudwatch logs.
*/
awsRegion: string;

/**
* Log Group Name in cloudwatch
*/
logGroupName: string

/**
* Prefix for logs stream
*/
log_stream_prefix: string;
elamaran11 marked this conversation as resolved.
Show resolved Hide resolved

/**
* Enable logs from fluentBit process
*/
enableFlbProcessLogs?: boolean
}

/**
* Default props for the add-on.
*/
const defaultProps: FluentBitConfigMapProps = {
awsRegion: "us-east-1",
logGroupName: "fargate-observability",
log_stream_prefix: "from-fluent-bit-",
elamaran11 marked this conversation as resolved.
Show resolved Hide resolved
enableFlbProcessLogs: false
};

/**
* Creates 'aws-observability' namespace and configurable ConfigMap
* to enable the Fargate built-in log router based on Fluent Bit
* https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html
*/
export class FluentBitConfigMap implements blueprints.ClusterAddOn {
Howlla marked this conversation as resolved.
Show resolved Hide resolved
id?: string | undefined;
readonly props: FluentBitConfigMapProps;

constructor(props?: FluentBitConfigMapProps) {
this.props = { ...defaultProps, ...props };
}

deploy(clusterInfo: blueprints.ClusterInfo): void {
const cluster = clusterInfo.cluster;

Check warning on line 56 in lib/single-new-eks-fargate-opensource-observability-pattern/fluentbitconfigmap.ts

View workflow job for this annotation

GitHub Actions / build (18)

'cluster' is assigned a value but never used

const doc = readYamlDocument(__dirname + '/../common/resources/fluent-bit/fluent-bit-fargate-config.ytpl');
const manifest = doc.split("---").map(e => loadYaml(e));

const values: blueprints.Values = {
awsRegion: this.props.awsRegion,
logGroupName: this.props.logGroupName,
log_stream_prefix: this.props.log_stream_prefix,
enableFlbProcessLogs: this.props.enableFlbProcessLogs,
};

const manifestDeployment: ManifestDeployment = {
name: 'aws-logging',
namespace: 'aws-observability',
manifest,
values
};

const kubectlProvider = new KubectlProvider(clusterInfo);
kubectlProvider.addManifest(manifestDeployment);

}
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,12 @@ import { Construct } from 'constructs';
import { utils } from '@aws-quickstart/eks-blueprints';
import * as blueprints from '@aws-quickstart/eks-blueprints';
import { GrafanaOperatorSecretAddon } from '../single-new-eks-opensource-observability-pattern/grafanaoperatorsecretaddon';
import { FluentBitConfigMap, FluentBitConfigMapProps } from './fluentbitconfigmap';
import * as amp from 'aws-cdk-lib/aws-aps';
import { ObservabilityBuilder } from '@aws-quickstart/eks-blueprints';
import * as eks from 'aws-cdk-lib/aws-eks';
import * as fs from 'fs';
import { ManagedPolicy,Role,ServicePrincipal } from "aws-cdk-lib/aws-iam";

export default class SingleNewEksFargateOpenSourceObservabilityConstruct {
constructor(scope: Construct, id: string) {
Expand Down Expand Up @@ -111,11 +113,17 @@ export default class SingleNewEksFargateOpenSourceObservabilityConstruct {
);
}

const fluentBitConfigMapProps = {
awsRegion: region,
logGroupName: "fargate-observability",
log_stream_prefix: "from-fluent-bit-",
elamaran11 marked this conversation as resolved.
Show resolved Hide resolved
} as FluentBitConfigMapProps;

Reflect.defineMetadata("ordered", true, blueprints.addons.GrafanaOperatorAddon);
const addOns: Array<blueprints.ClusterAddOn> = [
new blueprints.addons.VpcCniAddOn(),
new blueprints.addons.CoreDnsAddOn({
version: "v1.10.1-eksbuild.1",
version: "v1.10.1-eksbuild.6",
configurationValues: { computeType: "Fargate" }
}),
new blueprints.addons.KubeProxyAddOn(),
Expand All @@ -128,10 +136,6 @@ export default class SingleNewEksFargateOpenSourceObservabilityConstruct {
}),
new blueprints.addons.KubeStateMetricsAddOn(),
new blueprints.addons.MetricsServerAddOn(),
new blueprints.addons.CloudWatchLogsAddon({
logGroupPrefix: `/aws/eks/${stackId}`,
logRetentionDays: 30
}),
new blueprints.addons.ExternalsSecretsAddOn({
namespace: "external-secrets",
values: { webhook: { port: 9443 } }
Expand All @@ -141,9 +145,17 @@ export default class SingleNewEksFargateOpenSourceObservabilityConstruct {
new GrafanaOperatorSecretAddon(),
new blueprints.addons.AdotCollectorAddOn(),
new blueprints.addons.XrayAdotAddOn(),
new blueprints.addons.AmpAddOn(ampAddOnProps)
new blueprints.addons.AmpAddOn(ampAddOnProps),
new FluentBitConfigMap(fluentBitConfigMapProps)
];

const nodeRole = new blueprints.CreateRoleProvider("blueprint-fargate-pod-role", new ServicePrincipal("eks-fargate-pods.amazonaws.com"),
[
ManagedPolicy.fromAwsManagedPolicyName("AmazonEKSFargatePodExecutionRolePolicy"),
ManagedPolicy.fromAwsManagedPolicyName("CloudWatchAgentServerPolicy"),
]);

const podExecutionRole = blueprints.getNamedResource("blueprint-fargate-pod-role") as Role;

const fargateProfiles: Map<string, eks.FargateProfileOptions> = new Map([
["MyProfile", {
Expand All @@ -153,22 +165,24 @@ export default class SingleNewEksFargateOpenSourceObservabilityConstruct {
{ namespace: "external-secrets" },
{ namespace: "grafana-operator" },
{ namespace: "flux-system" }
]
}]
], podExecutionRole : podExecutionRole
}],
]);

// Define fargate cluster provider and pass the profile options
const fargateClusterProvider: blueprints.FargateClusterProvider = new blueprints.FargateClusterProvider({
fargateProfiles,
version: eks.KubernetesVersion.of("1.27")
version: eks.KubernetesVersion.of("1.28"),
});


ObservabilityBuilder.builder()
.account(account)
.region(region)
.clusterProvider(fargateClusterProvider)
.resourceProvider("blueprint-fargate-pod-role", nodeRole)
.resourceProvider(ampWorkspaceName, new blueprints.CreateAmpProvider(ampWorkspaceName, ampWorkspaceName))
.addOns(...addOns)
.build(scope, stackId);
}
}
}
Loading