PR #104265 - util/log: MVP of customizable 'datetime' field for JSON output

[cockroach-teamcity]

Exalate commented:

Related PR: cockroachdb/cockroach#104265
Commit: cockroachdb/cockroach@56da2cb
Fixes: CRDB-5594 CRDB-19901

---

Release note (cli change): The configuration for log output sinks now
accepts a new format-options field. This can be used to customize
the output of a given format. Each format accepts different options.

One available option for the json output format is
datetime-format. For example:

sinks:
  fluent-groups:
     custom-json:
        format: json
	format-options: {datetime-format: rfc3339}

This introduces a (new) field datetime in each output JSON event,
with the format specified by the option. As of this writing, the
following values are documented:

• none: disable the creation of the datetime field.
  This is the default value.
• iso8601 / rfc3339: format the time stamp like
  "2006-01-02T15:04:05.999999999Z".
• rfc1123: format the time stamp like "Mon, 02 Jan 2006 15:04:05 +0000".

Note that enabling the datetime field introduces CPU overhead
and is nearly always unnecessary. When using output to a log
collector (e.g. via Fluent, Datadog etc) the log collector can
be configured to transform the timestamp provided by CockroachDB,
without requiring participation from CockroachDB itself. When
inspecting a log file containing JSON output produced by CockroachDB,
the command cockroach debug merge-log can be used to consume the
JSON data and reformat it using the crdb-v2 format which also
includes the date/time using the RFC3339 format.

Jira Issue: DOC-8095

[exalate-issue-sync]

Tommy Truongchau (thtruo) commented:
cc Michael Lewis this one doesn’t need to be prioritized in the August milestone, but I do want to flag a few things:

this enhancement is part of the 23.2 release, and should be documented before that release as it’s part of a customer commit with Fidelity giving them a way to set timezones for logs

that this PR touches multiple DOC issues (so perhaps for ease of tracking, either we move all these issues into the same sprint, or close out the rest while keeping this issue as the tracking item - will defer to you the best way to manage it from the docs perspective):

https://cockroachlabs.atlassian.net/browse/DOC-8098

https://cockroachlabs.atlassian.net/browse/DOC-8099

https://cockroachlabs.atlassian.net/browse/DOC-8100

https://cockroachlabs.atlassian.net/browse/DOC-8101

https://cockroachlabs.atlassian.net/browse/DOC-8102

https://cockroachlabs.atlassian.net/browse/DOC-8103

https://cockroachlabs.atlassian.net/browse/DOC-8104

[exalate-issue-sync]

Florence Morris (florence-crl) commented:
Bug with json format options, see slack thread

Alex Barganier

{quote}Aha! Definitely a bug.Despite saying that it's {{format: json}} - it's using {{format: crdb-v2}}! Because it's confusing the formats, it's using the wrong parser (the crdb-v2 one) when trying to interpret the format options. Of course, {{datetime-format}} isn't supported in crdb-v2, so it gives up and errors out.Glad you brought this to our attention! I'll work on a fix. Thanks Florence{quote}

[exalate-issue-sync]

Florence Morris (florence-crl) commented:
GH issue: cockroachdb/cockroach#113321

[exalate-issue-sync]

Florence Morris (florence-crl) commented:
PR with a fix on master: cockroachdb/cockroach#113532

PR backported to 23.2: cockroachdb/cockroach#113684