Merge pull request #107 from aws-observability/log-docs

Log docs
aws-observability · Aug 30, 2023 · 7e8283b · 7e8283b
2 parents a119c0a + bbffe14
commit 7e8283b
Show file tree

Hide file tree

Showing 7 changed files with 80 additions and 137 deletions.
diff --git a/docs/logs.md b/docs/logs.md
@@ -0,0 +1,74 @@
+# Viewing Logs
+
+By default, we deploy a FluentBit daemon set in the cluster to collect worker logs for all namespaces. Logs are collected and exported to Amazon CloudWatch Logs, which enables you to centralize the logs from all of your systems, applications, and AWS services that you use, in a single, highly scalable service.
+
+Further configuration options are available in the module documentation. This guide shows how you can leverage either CloudWatch Logs or Amazon Managed Grafana for your cluster and application logs.
+
+## Viewing Logs in CloudWatch Logs Insights
+
+Navigate to CloudWatch, then go to "Logs Insights"
+
+In the dropdown, select any of the logs that begin with "/aws/eks/single-new-eks-mixed-observability-accelerator" and run a query.
+
+Example with "kubesystem" log group:
+
+![logs-query](./patterns/images/mixed-query.png)
+
+Then you can view the results of your query:
+
+![logs-results](./patterns/images/mixed-log-results.png)
+
+## Viewing Logs in Grafana
+
+### Using CloudWatch Logs as data source in Grafana
+
+Follow [the documentation](https://docs.aws.amazon.com/grafana/latest/userguide/using-amazon-cloudwatch-in-AMG.html)
+to enable Amazon CloudWatch as a data source. Make sure to provide permissions.
+
+All logs are delivered in the following CloudWatch Log groups naming pattern:
+`/aws/eks/$PATTERN`.
+Log streams follow `{container-name}.{pod-name}`. In Grafana, querying and analyzing logs is done with [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html)
+
+### Example - ADOT collector logs
+
+Select one or many log groups and run the following query. The example below,
+queries AWS Distro for OpenTelemetry (ADOT) logs
+
+```console
+fields @timestamp, log
+| order @timestamp desc
+| limit 100
+```
+
+![logs-1](./patterns/images/logs-1.png)
+
+### Example - Using time series visualizations
+
+[CloudWatch Logs syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html)
+provide powerful functions to extract data from your logs. The `stats()`
+function allows you to calculate aggregate statistics with log field values.
+This is useful to have visualization on non-metric data from your applications.
+
+In the example below, we use the following query to graph the number of metrics
+collected by the ADOT collector
+
+```console
+fields @timestamp, log
+| parse log /"#metrics": (?<metrics_count>\d+)}/
+| stats avg(metrics_count) by bin(5m)
+| limit 100
+```
+
+!!! tip
+    You can add logs in your dashboards with logs panel types or time series
+    depending on your query results type.
+
+![logs-2](./patterns/images/logs-2.png)
+
+!!! warning
+    Querying CloudWatch logs will incur costs per GB scanned. Use small time
+    windows and limits in your queries. Checkout the CloudWatch
+    [pricing page](https://aws.amazon.com/cloudwatch/pricing/) for more info.
+
+
+
diff --git a/...existing-eks-observability-accelerators/existing-eks-awsnative-observability.md b/...existing-eks-observability-accelerators/existing-eks-awsnative-observability.md
@@ -102,7 +102,7 @@ prometheus-node-exporter   Active   13m
 - On a same view, select 'EKS Pods', which provides insights overview of all the pods as shown below
 ![Container Insights - EKS Pods](../images/awsnative-container-insights-2.png)
 
-- Please see [Single New EKS Cluster AWS Mixed Observability Accelerator](../single-new-eks-observability-accelerators/single-new-eks-mixed-observability.md#viewing-logs) for Logs.
+- Refer to "Using CloudWatch Logs Insights to Query Logs in [Logging](../../logs.md).
 
 ## Teardown
 

diff --git a/...le-new-eks-observability-accelerators/single-new-eks-awsnative-observability.md b/...le-new-eks-observability-accelerators/single-new-eks-awsnative-observability.md
@@ -113,22 +113,7 @@ View the Performance Monitoring Dashboard:
 
 ## Viewing Logs
 
-By default, we deploy a FluentBit daemon set in the cluster to collect worker logs for all namespaces. Logs are collected and exported to Amazon CloudWatch Logs, which enables you to centralize the logs from all of your systems, applications,
-and AWS services that you use, in a single, highly scalable service.
-
-## Using CloudWatch Logs Insights to Query Logs
-
-Navigate to CloudWatch, then go to "Logs Insights"
-
-In the dropdown, select any of the logs that begin with "/aws/eks/single-new-eks-awsnative-observability-accelerator" and run a query.
-
-Example with "kubesystem" log group:
-
-![logs-query](../images/logs-insights1.png)
-
-Then you can view the results of your query:
-
-![logs-results](../images/logs-insights2.png)
+Refer to "Using CloudWatch Logs Insights to Query Logs in [Logging](../../logs.md).
 
 
 ## Teardown

diff --git a/...-observability-accelerators/single-new-eks-graviton-opensource-observability.md b/...-observability-accelerators/single-new-eks-graviton-opensource-observability.md
@@ -290,58 +290,7 @@ Grafana Operator and Flux always work together to synchronize your dashboards wi
 
 ## Viewing Logs
 
-By default, we deploy a FluentBit daemon set in the cluster to collect worker logs for all namespaces. Logs are collected and exported to Amazon CloudWatch Logs, which enables you to centralize the logs from all of your systems, applications,
-and AWS services that you use, in a single, highly scalable service.
-
-## Using CloudWatch Logs as data source in Grafana
-
-Follow [the documentation](https://docs.aws.amazon.com/grafana/latest/userguide/using-amazon-cloudwatch-in-AMG.html)
-to enable Amazon CloudWatch as a data source. Make sure to provide permissions.
-
-All logs are delivered in the following CloudWatch Log groups naming pattern:
-`/aws/eks/single-new-eks-opensource-observability-accelerator`.
-Log streams follow `{container-name}.{pod-name}`. In Grafana, querying and analyzing logs is done with [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html)
-
-### Example - ADOT collector logs
-
-Select one or many log groups and run the following query. The example below,
-queries AWS Distro for OpenTelemetry (ADOT) logs
-
-```console
-fields @timestamp, log
-| order @timestamp desc
-| limit 100
-```
-
-![logs-1](../images/logs-1.png)
-
-### Example - Using time series visualizations
-
-[CloudWatch Logs syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html)
-provide powerful functions to extract data from your logs. The `stats()`
-function allows you to calculate aggregate statistics with log field values.
-This is useful to have visualization on non-metric data from your applications.
-
-In the example below, we use the following query to graph the number of metrics
-collected by the ADOT collector
-
-```console
-fields @timestamp, log
-| parse log /"#metrics": (?<metrics_count>\d+)}/
-| stats avg(metrics_count) by bin(5m)
-| limit 100
-```
-
-!!! tip
-    You can add logs in your dashboards with logs panel types or time series
-    depending on your query results type.
-
-![logs-2](../images/logs-2.png)
-
-!!! warning
-    Querying CloudWatch logs will incur costs per GB scanned. Use small time
-    windows and limits in your queries. Checkout the CloudWatch
-    [pricing page](https://aws.amazon.com/cloudwatch/pricing/) for more infos.
+Refer to the "Using CloudWatch Logs as a data source in Grafana" section in [Logging](../../logs.md).
 
 ## Teardown
 

diff --git a/...single-new-eks-observability-accelerators/single-new-eks-mixed-observability.md b/...single-new-eks-observability-accelerators/single-new-eks-mixed-observability.md
@@ -106,22 +106,7 @@ View the graph of the selected metrics:
 
 ## Viewing Logs
 
-By default, we deploy a FluentBit daemon set in the cluster to collect worker logs for all namespaces. Logs are collected and exported to Amazon CloudWatch Logs, which enables you to centralize the logs from all of your systems, applications,
-and AWS services that you use, in a single, highly scalable service.
-
-## Using CloudWatch Logs Insights to Query Logs
-
-Navigate to CloudWatch, then go to "Logs Insights"
-
-In the dropdown, select any of the logs that begin with "/aws/eks/single-new-eks-mixed-observability-accelerator" and run a query.
-
-Example with "kubesystem" log group:
-
-![logs-query](../images/mixed-query.png)
-
-Then you can view the results of your query:
-
-![logs-results](../images/mixed-log-results.png)
+Refer to "Using CloudWatch Logs Insights to Query Logs in [Logging](../../logs.md).
 
 ## Teardown
 

diff --git a/...e-new-eks-observability-accelerators/single-new-eks-opensource-observability.md b/...e-new-eks-observability-accelerators/single-new-eks-opensource-observability.md
@@ -246,58 +246,7 @@ Grafana Operator and Flux always work together to synchronize your dashboards wi
 
 ## Viewing Logs
 
-By default, we deploy a FluentBit daemon set in the cluster to collect worker logs for all namespaces. Logs are collected and exported to Amazon CloudWatch Logs, which enables you to centralize the logs from all of your systems, applications,
-and AWS services that you use, in a single, highly scalable service.
-
-## Using CloudWatch Logs as data source in Grafana
-
-Follow [the documentation](https://docs.aws.amazon.com/grafana/latest/userguide/using-amazon-cloudwatch-in-AMG.html)
-to enable Amazon CloudWatch as a data source. Make sure to provide permissions.
-
-All logs are delivered in the following CloudWatch Log groups naming pattern:
-`/aws/eks/single-new-eks-opensource-observability-accelerator`.
-Log streams follow `{container-name}.{pod-name}`. In Grafana, querying and analyzing logs is done with [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html)
-
-### Example - ADOT collector logs
-
-Select one or many log groups and run the following query. The example below,
-queries AWS Distro for OpenTelemetry (ADOT) logs
-
-```console
-fields @timestamp, log
-| order @timestamp desc
-| limit 100
-```
-
-![logs-1](../images/logs-1.png)
-
-### Example - Using time series visualizations
-
-[CloudWatch Logs syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html)
-provide powerful functions to extract data from your logs. The `stats()`
-function allows you to calculate aggregate statistics with log field values.
-This is useful to have visualization on non-metric data from your applications.
-
-In the example below, we use the following query to graph the number of metrics
-collected by the ADOT collector
-
-```console
-fields @timestamp, log
-| parse log /"#metrics": (?<metrics_count>\d+)}/
-| stats avg(metrics_count) by bin(5m)
-| limit 100
-```
-
-!!! tip
-    You can add logs in your dashboards with logs panel types or time series
-    depending on your query results type.
-
-![logs-2](../images/logs-2.png)
-
-!!! warning
-    Querying CloudWatch logs will incur costs per GB scanned. Use small time
-    windows and limits in your queries. Checkout the CloudWatch
-    [pricing page](https://aws.amazon.com/cloudwatch/pricing/) for more infos.
+Refer to the "Using CloudWatch Logs as a data source in Grafana" section in [Logging](../../logs.md).
 
 ## Teardown
 

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -35,6 +35,7 @@ nav:
       - OSS Java Mon : patterns/single-new-eks-observability-accelerators/single-new-eks-java-opensource-observability.md
       - Mixed : patterns/single-new-eks-observability-accelerators/single-new-eks-mixed-observability.md
       - Graviton OSS : patterns/single-new-eks-observability-accelerators/single-new-eks-graviton-opensource-observability.md
+    - Logs: logs.md  
     - Tracing: tracing.md
   - Supporting Examples:
     - EKS Cluster: patterns/single-new-eks-observability-accelerators/single-new-eks-cluster.md