Feature: Integrate custom metrics with new CloudWatch embedded metric

[heitorlessa]

Currently, we require an additional log processing stack to create custom metrics. Amazon CloudWatch just released support for high cardinality and custom metrics with a new metric format that we could integrate to.

Tasks:

• Read CloudWatch metrics Python library
• Measure the value of adhering to the format vs bringing an additional dependency

[heitorlessa]

PR implementing Lambda support in new CW Library: awslabs/aws-embedded-metrics-python@a352d40

At a first glance, it looks like it worth bringing as an additional dependency as it handles a few edge cases w/ dimensions, captures Lambda context including trace ID, etc.

Pros

• Captures Lambda context as high-cardinality data
• Handles dimensions and other limits

Cons

• Another dependency
• metric_scope decorator doesn't take into account outer functions where metrics could also be generated
• On a second look, we could implement a serializer and get away with a simpler interface

Will put it to test, looking at the Lambda specific implementation a Serializer and a print could do.

[heitorlessa]

Chatting with @nmoutschen, it's possible to keep a simpler interface like we have with log_metric but change the implementation to keep metrics in state and flush them upon handler return.

Initial idea to try possibly next week:

• Keep the simpler to use log_metric(name, value, unit, dimensions, etc.)
• Change log_metric to instead of print keep state
• [Optional] Create a decorator for lambda to flush metrics
  • I'd prefer flushing metric immediately as fn can crash and it'd lose all state
• [Optional] Flush metrics once it reaches 20 metrics - may not be needed, need to read the docs and implementation some more

---

Working example - Gotcha here is that timestamp must be in ms

import datetime
import json

def lambda_handler(event, context):
    print(json.dumps({
        "_aws": {
            "CloudWatchMetrics": [
                {
                    "Namespace": "Test/CustomMetrics",
                    "Dimensions": [["functionVersion"]],
                    "Metrics": [
                        {
                            "Name": "time",
                            "Unit": "Milliseconds"
                        }
                    ],
                }
            ],
            "Timestamp": int(datetime.datetime.now().timestamp()*1000)
        },
        "functionVersion": context.function_version,
        "time": 100,
        "requestId": context.aws_request_id
    }))

[heitorlessa]

Another discovery that I overlooked at first - Dimensions are shared across all metrics within the array to be flushed. That could easily bring costs up if one is not aware of that since each dimension creates a new metric.

[heitorlessa]

First implementation -> https://gist.github.com/heitorlessa/5c918d35073bc4c7223de7ffdcc18735

I'll port to the Serverless Airline project first that still uses the original version of the powertools. Then, I'll create a PR and fully document it.

Leaving it here in case someone needs it