Feature request: logger should write to stdout directly

[olanb7]

Logger should write to stdout, and not use console

Problem statement

The powertools logger is currently outputting it's logs via console Logger.ts#L451. Lambda appears to use a patched version of console which prefixes the timestamp, requestId and log level. This means that a powertools logged event is output to stdout as below, which is clearly not JSON:

2022-04-08T10:58:44.811Z	b6511ee9-4873-404e-b60c-a23cb45d3ff2	INFO {"cold_start":false,"function_arn":"arn:aws:lambda:eu-west-2:12345:function:test","function_memory_size":1024,"function_name":"test","function_request_id":"b6511ee9-4873-404e-b60c-a23cb45d3ff2","level":"INFO","message":"test message","service":"test","timestamp":"2022-04-08T10:58:44.811Z"}

Cloudwatch handles this OK, but other tools like the newrelic and honeycomb extensions work under the assumption that JSON logs are being sent to stdout. Honeycomb even warns against the use of console.log in their documentation.

Following the honeycomb suggestion of using bunyan (which does write to stdout) I have validated (as has @dreamorosi on slack) that these prefixes can be avoided by bypassing console.log.

The information in these prefixes is also duplicated already in the JSON message itself, so there is no loss of querying (although some may prefer to keep these prefixes so that cloudwatch queries aren't impacted.

Summary of the feature

Update the logger to avoid the patched console.log by writing logs to stdout. This will ensure that valid JSON messages can be intercepted by lambda layers and easily forwarded on to other observability tools.

Benefits for you and the wider AWS community

This is a blocker right now to anyone using centralised logging tools like NewRelic or Honeycomb, for whom structured logs are critical to the maintenance and support.

Describe alternatives you've considered

The only alternative is to use a different logger like pino or bunyan.

Additional context

I have not been able to find the source for the lambda logger internals (presumably closed source) but have verified that it must be adding the prefixes by bypassing.

Related issues, RFCs

#646
#lambda-powertools slack discussion

[saragerion]

Thanks a lot of opening this issue. We want developers to be able to send their logs to external centralised monitoring tools, and for that they need to be parsed correctly as valid JSON not only by CloudWatch.

As part of the unit of work of this issue we could consider also adding a E2E tests that validate the output of the logs as valid JSON.

[dreamorosi]

We were able to come up with a fix and avoid having the prefix on the logs by overriding the global Console object.

I'm working on a PR to fix this and have already applied the change, but it'll take me a while to adapt our tests to reflect the changes.

You can track progress in #748.

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[dreamorosi]

The change has been merged in #748 and will be released in the next version.