diff --git a/docs/patterns/fluentbit-containerinsights.md b/docs/patterns/fluentbit-containerinsights.md new file mode 100644 index 0000000000..53b65f5f11 --- /dev/null +++ b/docs/patterns/fluentbit-containerinsights.md @@ -0,0 +1,7 @@ +--- +title: AWS for FluentBit with ContainerInsights +--- + +{% + include-markdown "../../patterns/fluentbit-containerinsights/README.md" +%} diff --git a/patterns/fluentbit-containerinsights/README.md b/patterns/fluentbit-containerinsights/README.md new file mode 100644 index 0000000000..d180513755 --- /dev/null +++ b/patterns/fluentbit-containerinsights/README.md @@ -0,0 +1,337 @@ +# AWS for FluentBit with ContainerInsights and Kubelet monitoring. + +This pattern demonstrates an Amazon EKS Cluster deployment using AWS for FluentBit integration with ContainerInsights and Kubelet monitoring to send logs to Amazon CloudWatch Logs. + +- [Container Insights on Amazon EKS and Kubernetes](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-EKS-quickstart.html#Container-Insights-setup-EKS-quickstart-FluentBit) +- [Use_Kubelet feature](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-use-kubelet.html) + +## Areas of Interest + +Here is the configuration block under the `eks_blueprints_addons` module that customizes the AWS for FluentBit addon in order to enable ContainerInsights and Kubelet monitoring feature. + +https://github.com/rodrigobersa/terraform-aws-eks-blueprints/blob/97ea696788fcace8d5fa0ee91927efd4b8a549df/patterns/fluentbit-containerinsights/main.tf#L99-L125 + +## Deploy + +See [here](https://aws-ia.github.io/terraform-aws-eks-blueprints/getting-started/#prerequisites) for the prerequisites and steps to deploy this pattern. + +## Validate + +1. List all the Pods in the cluster; you should see `aws-for-fluentbit-` Pods running on `kube-system` Namespace. All the Pods should reach a status of `Running` after approximately 60 seconds: + +```bash +$ kubectl get pods -A +``` +```bash +NAMESPACE NAME READY STATUS RESTARTS AGE +amazon-guardduty aws-guardduty-agent-2bkb8 1/1 Running 0 4m28s +amazon-guardduty aws-guardduty-agent-mdj2q 1/1 Running 0 4m28s +amazon-guardduty aws-guardduty-agent-vxkvg 1/1 Running 0 4m28s +kube-system aws-for-fluent-bit-54lmh 1/1 Running 0 4m34s +kube-system aws-for-fluent-bit-89wbv 1/1 Running 0 4m33s +kube-system aws-for-fluent-bit-nt2qr 1/1 Running 0 4m34s +kube-system aws-node-q55f2 1/1 Running 0 4m25s +kube-system aws-node-tx5s8 1/1 Running 0 4m8s +kube-system aws-node-vlgpr 1/1 Running 0 4m16s +kube-system coredns-6c45d94f67-dvmvf 1/1 Running 0 8m51s +kube-system coredns-6c45d94f67-xfb97 1/1 Running 0 8m51s +kube-system kube-proxy-5qcdm 1/1 Running 0 4m42s +kube-system kube-proxy-9hksp 1/1 Running 0 4m41s +kube-system kube-proxy-c2hcx 1/1 Running 0 4m42s +kube-system metrics-server-8794b9cdf-l7hzq 1/1 Running 0 6m8s +``` + +2. Validate the `aws-for-fluent-bit` configMap was created correctly in the `kube-system` Namespace. Make sure that the configMap has the `application-log.conf`, `dataplane-log.conf`, `host-log.conf`, and `fluent-bit.conf` definitions. Also, check the parameters `Use_Kubelet On` and `Kubelet_Port 10250` are set! + +```bash +$ kubectl get cm -n kube-system aws-for-fluent-bit -o yaml +``` +```yaml +apiVersion: v1 +data: + application-log.conf: | + [INPUT] + Name tail + Tag application.* + Exclude_Path /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy* + Path /var/log/containers/*.log + multiline.parser docker, cri + DB /var/fluent-bit/state/flb_container.db + Mem_Buf_Limit 50MB + Skip_Long_Lines On + Refresh_Interval 10 + Rotate_Wait 30 + storage.type filesystem + Read_from_Head Off + + [INPUT] + Name tail + Tag application.* + Path /var/log/containers/fluent-bit* + multiline.parser docker, cri + DB /var/fluent-bit/state/flb_log.db + Mem_Buf_Limit 5MB + Skip_Long_Lines On + Refresh_Interval 10 + Read_from_Head Off + + [INPUT] + Name tail + Tag application.* + Path /var/log/containers/cloudwatch-agent* + multiline.parser docker, cri + DB /var/fluent-bit/state/flb_cwagent.db + Mem_Buf_Limit 5MB + Skip_Long_Lines On + Refresh_Interval 10 + Read_from_Head Off + + [FILTER] + Name kubernetes + Match application.* + Kube_URL https://kubernetes.default.svc:443 + Kube_Tag_Prefix application.var.log.containers. + Merge_Log On + Merge_Log_Key log_processed + K8S-Logging.Parser On + K8S-Logging.Exclude Off + Labels Off + Annotations Off + Use_Kubelet On + Kubelet_Port 10250 + Buffer_Size 0 + + [OUTPUT] + Name cloudwatch_logs + Match application.* + region us-west-2 + log_group_name /aws/containerinsights/fluentbit-containerinsights/application + log_stream_prefix ${HOSTNAME}- + auto_create_group true + extra_user_agent container-insights + workers 1 + dataplane-log.conf: | + [INPUT] + Name systemd + Tag dataplane.systemd.* + Systemd_Filter _SYSTEMD_UNIT=docker.service + Systemd_Filter _SYSTEMD_UNIT=containerd.service + Systemd_Filter _SYSTEMD_UNIT=kubelet.service + DB /var/fluent-bit/state/systemd.db + Path /var/log/journal + Read_From_Tail On + + [INPUT] + Name tail + Tag dataplane.tail.* + Path /var/log/containers/aws-node*, /var/log/containers/kube-proxy* + multiline.parser docker, cri + DB /var/fluent-bit/state/flb_dataplane_tail.db + Mem_Buf_Limit 50MB + Skip_Long_Lines On + Refresh_Interval 10 + Rotate_Wait 30 + storage.type filesystem + Read_from_Head Off + + [FILTER] + Name modify + Match dataplane.systemd.* + Rename _HOSTNAME hostname + Rename _SYSTEMD_UNIT systemd_unit + Rename MESSAGE message + Remove_regex ^((?!hostname|systemd_unit|message).)*$ + + [FILTER] + Name aws + Match dataplane.* + imds_version v2 + + [OUTPUT] + Name cloudwatch_logs + Match dataplane.* + region us-west-2 + log_group_name /aws/containerinsights/fluentbit-containerinsights/dataplane + log_stream_prefix ${HOSTNAME}- + auto_create_group true + extra_user_agent container-insights + fluent-bit.conf: | + [SERVICE] + Flush 5 + Grace 30 + Log_Level info + Daemon off + Parsers_File parsers.conf + HTTP_Server On + HTTP_Listen 0.0.0.0 + HTTP_Port 2020 + storage.path /var/fluent-bit/state/flb-storage/ + storage.sync normal + storage.checksum off + storage.backlog.mem_limit 5M + + @INCLUDE application-log.conf + @INCLUDE dataplane-log.conf + @INCLUDE host-log.conf + host-log.conf: | + [INPUT] + Name tail + Tag host.dmesg + Path /var/log/dmesg + Key message + DB /var/fluent-bit/state/flb_dmesg.db + Mem_Buf_Limit 5MB + Skip_Long_Lines On + Refresh_Interval 10 + Read_from_Head Off + + [INPUT] + Name tail + Tag host.messages + Path /var/log/messages + Parser syslog + DB /var/fluent-bit/state/flb_messages.db + Mem_Buf_Limit 5MB + Skip_Long_Lines On + Refresh_Interval 10 + Read_from_Head Off + + [INPUT] + Name tail + Tag host.secure + Path /var/log/secure + Parser syslog + DB /var/fluent-bit/state/flb_secure.db + Mem_Buf_Limit 5MB + Skip_Long_Lines On + Refresh_Interval 10 + Read_from_Head Off + + [FILTER] + Name aws + Match host.* + imds_version v2 + + [OUTPUT] + Name cloudwatch_logs + Match host.* + region us-west-2 + log_group_name /aws/containerinsights/fluentbit-containerinsights/host + log_stream_prefix ${HOSTNAME}. + auto_create_group true + extra_user_agent container-insights + parsers.conf: | + [PARSER] + Name syslog + Format regex + Regex ^(?