Documentation on using custom date format for logging

[kishaningithub]

What were you initially searching for in the docs?
I was searching for a way to use ISO8601 especially the RFC3339 variant while printing the event time stamps. Then i found this discussion link #905 (comment)

Is this related to an existing part of the documentation? Please share a link
https://awslabs.github.io/aws-lambda-powertools-python/latest/core/logger/#

Describe how we could make it clearer
I would be really helpful if we can provide an example of how to change the timeformat to RFC3339 in the documentation site here

If you have a proposed update, please share it here
Example on how to instantiate logger using RFC3339 format

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.

[heitorlessa]

Hey @kishaningithub - have you seen this part of the docs? If so, was it helpful?

https://awslabs.github.io/aws-lambda-powertools-python/latest/core/logger/#overriding-log-records

We should totally make a dedicated section to esse discovering either way - thank you so much for helping us improve the docs!

[kishaningithub]

@heitorlessa Thanks for pointing me to this section of the documentation. Originally I had skipped this section (viz "Migrating from other loggers") because i was creating a new project from scratch. This section is definitely useful :-)

Given RFC3339 is such a common thing now, for ease of use i am also thinking why not create a helper method which returns the datefmt string for this format. This can be used while instantiating the logger.

[heitorlessa]

Great idea! Would you like to contribute it as a static method within Logger? Source class: https://github.com/awslabs/aws-lambda-powertools-python/blob/3e4e1abaebea360c01752726b62af8b805bbe3ac/aws_lambda_powertools/logging/logger.py#L48 I'd be more than happy to review/guide. If you can't, we can tackle after reviewing the new tutorial docs section and MyPy fixes

…

On Tue, 11 Jan 2022 at 11:03, Kishan B ***@***.***> wrote: @heitorlessa <https://github.com/heitorlessa> Thanks for pointing me to this section of the documentation. Originally I had skipped this section (viz "Migrating from other loggers") because i was creating a new project from scratch. This section is definitely useful :-) Given RFC3339 is such a common thing now, for ease of use i am also thinking why not create a helper method which returns the datefmt string for this format. This can be used while instantiating the logger. — Reply to this email directly, view it on GitHub <#946 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBBVWIOVLXT26Z3EFCLUVP575ANCNFSM5LVNBDXA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[kishaningithub]

I will do it @heitorlessa Thanks for your support

[heitorlessa]

hey @kishaningithub following up on this :) An efficient and extensible way to do this could be as constants. That way, it could be easier to document, keep Logger clean, allow reusing for any other loggers in the future, and allow future contributions with variations of it.

e.g. pseudo code, doesn't have to be this way

app.py

from aws_lambda_powertools.logging import date_formats

logger = Logger(datefmt=date_formats.RFC3339)
logger = Logger(datefmt=date_formats.ISO8601)

aws_lambda_powertools/logging/date_formats.py

RFC3339 = "<format>"
ISO8601 = "<format>"
# any variation, or formats that might require datetime str 

[heitorlessa]

hey @kishaningithub following up to confirm you still have the bandwidth to make this contribution - if you do, please do not hesitate to ask for help (here or on our discord)

[heitorlessa]

Assigning this to myself to make it available in the next release @kishaningithub @JJSphar

I've had a head-scratching moment when trying to find the differences between ISO8601 and RFC3339 (an extension/profile of ISO8601).

After a lot of trial and error, luckily, someone built a tool to isolate with formats that show the interception and where they don't agree with each other.

Here's a breakdown of challenges and decisions so we can always come back to this later if needed:

• Challenge 1: The differences are subtle like a space between date and timestamp, or TZ differences. [Solution] Use one format only that is valid for both specs: 2022-10-27T16:27:43.738+02:00
• Challenge 2: Python logging by default uses time.strftime formats TZ slightly different breaking RFC3339 (+0100 vs +01:00). [Solution] Create a new flag in PowertoolsFormatter to use both the correct date format and use_datetime_directive=True, instead of an Enum-like I proposed earlier. This will prevent mistakes when someone tries to use that date format string but forgets to use datetime breaking operational/analytical tools later.

Quick representation of default and this new option

Type	Example	Format
Default in Logger	2022-10-27 16:14:22,386+0200	%Y-%m-%d %H:%M:%S,%F%z
RFC3339_ISO8601	2022-10-27T16:27:43.738+02:00	%Y-%m-%dT%H:%M:%S.%F%z

With UTC: 2022-10-27T14:39:42.728+0000

UX

from aws_lambda_powertools import Logger

logger = Logger(use_rfc3339_iso8601=True)

[kishaningithub]

In my opinion RFC3339 is a stricter version of ISO8601.

Golang's std library only supports RFC3339 which feels like a right choice to me to apply in other places

[JJSphar]

This sounds great! I think the format you listed would cover a lot of common use cases.

Personally, my lambdas use a wrapper class around Logger that handles setting a custom format and the use_datetime_directive arg. Being able to output in RFC3339 directly from Logger without having to copy/paste that format string around will let me get rid of this wrapper class. Thanks!

[heitorlessa]

Draft PR is up: #1662

@kishaningithub @JJSphar could I please ask you to double check whether that format and UX address the original ask?

I'll document tomorrow if that suffice. I've added an internal note so that we move to that format in V3, add appropriate warnings in the future, and more importantly to not cause a breaking change.

Thank you for your patience.

[heitorlessa]

PR is now complete - Docs will now feature a dedicated section for Date formatting.

Final UX: Logger(use_rfc3339=True. Date format remains the same. It should be out in the next release either by EOD or Monday

[kishaningithub]

I like Logger(use_rfc3339=True UX much better than the previous one :-)

[heitorlessa]

yes, thank @leandrodamascena for that great suggestion :D

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

This is now released under 2.1.0 version!