[core.logging] Ensure LogMeta is ECS-compliant.

[lukeelmers]

Closes #90406

One goal of our new logging system is for the logs to be ECS-compliant (when using JSON layout). The one part of a log record that is outside of core's control is the LogMeta, where developers can put pretty much any object they want and have it added to the logs.

While this provides nice flexibility, it poses a challenge when it comes to adhering to ECS formats: since we merge any LogMeta into the actual log record itself, it means that folks would ideally place their meta into a known ECS field (if one exists for their data), or otherwise choose a custom field to use.

In looking more closely at #90406, we decided that we would create a full set of ECS typings in core to use for promoting ECS-friendly LogMeta.

This PR makes a few different changes to help with this effort:

1. Adds full set of ECS typings (v1.9) to @kbn/logging, and also exports them from core.
2. Updates LogMeta types to ensure they adhere to ECS types.
3. Adds generic type params to Logger so that folks can pass their own custom types which extend from thee base LogMeta/ECS types.
4. Updates downstream uses of LogMeta in core/plugins.
5. Makes a few changes to security's audit logging, which already had its own ECS implementation that we can now align with core's.

Plugin API Changes

Core's logging system has been updated to ensure that logs using a JSON layout are compliant with ECS. If you are using the optional LogMeta param in your plugin, you should check the ECS spec and ensure your meta conforms to ECS wherever possible.

To assist with this, we've updated the LogMeta TypeScript interface to require ECS-friendly data. Should you need to log fields that do not fit within the ECS spec, you can provide a generic type parameter which accepts an interface that extends from the base LogMeta:

// plugins/my-plugin/server/plugin.ts

import type { CoreSetup, LogMeta, Plugin, PluginInitializerContext } from 'src/core/server';

interface MyPluginLogMeta extends LogMeta {
  kibana: { myCustomField: string };
}

export class MyPlugin implements Plugin  {
  constructor(private readonly initContext: PluginInitializerContext) {
    this.logger = initContext.logger.get();
  }

  setup(core: CoreSetup) {
    this.logger.warn<MyPluginLogMeta>('my log with custom meta', {
      kibana: {
        myCustomField: 'heya',
      }
    });
  }
}

[lukeelmers]

I ended up exporting these from core because audit logging will use some of them.

[lukeelmers]

Spent awhile battling TS on this (and a similar situation in the actions plugin).

The issue is that dynamically calling a method from Logger results in an This expression is not callable error due to each member of the union type having signatures which are incompatible (similar to this issue).

Specifically, it seems that when you are mixing methods which could contain string | Error in the message (warn, error, fatal) with ones that only contain string (trace, debug, info), TS freaks out.

But interestingly, this only surfaced when I added the generic type params to the Logger interface... removing them resolves the issue.

Still not sure what's up with that, but would appreciate any insights folks may have. In the meantime, I resorted to casting in both places 😞

[lukeelmers]

Pinging @mshustov @thomheymann for any initial thoughts on the approach here before I take it out of draft and ping a bunch of people 🙂

I added a few comments for discussion.

[thomheymann]

Nice one, looks great Luke!

One suggestion regarding quarantined fields below.

[mshustov]

what about avoiding Omit and Pick in the public interfaces? Let's a public interface with ecs field internally

the current error message :

[lukeelmers]

Let's a public interface with ecs field internally

The only issue I can see with this is a maintenance one -- if we update versions of ECS (or if someday the types are auto generated), then we will need to manually update the top-level keys in LogMeta as well. However, since top-level keys don't change all that much, it probably wouldn't hurt to duplicate them for now, and reconsider later if it turns out to be a problem in practice. Will update.

[lukeelmers]

I tested the Kibana filebeat module and confirmed that converting this to an array, while correct from an ECS standpoint, will still break existing filters, etc:

I'm working on an update to the module to correct this.

[lukeelmers]

Draft PR for the filebeat module: elastic/beats#25101

(@legrego @thomheymann would appreciate any expertise you could offer on that) ❤️

[thomheymann]

Thanks for testing how this gets displayed. I think we might need to check with the Observability team if they support arrays for event.type and event.category and how that should be rendered.

[thomheymann]

Looks great, really happy with the improvements!

Tested locally and audit logging is working as expected.

Got two questions. One below regarding ignore filters and the other one in the link regarding backwards compatibility: https://github.com/elastic/beats/pull/25101/files#r613936133

[lukeelmers]

@elasticmachine merge upstream

[pmuellr]

alerting changes LGTM

[thomheymann]

Audit logging changed LGTM! 👍

[lukeelmers]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: a0a2bad
• Storybooks Preview

Metrics [docs]

Unknown metric groups

API count

id	before	after	diff
core	2174	2178	+4
security	79	71	-8
total			-4

API count missing comments

id	before	after	diff
core	992	995	+3
security	38	31	-7
total			-4

API count with any type

id	before	after	diff
core	147	146	-1

History

• 💚 Build #120159 succeeded 28ef538
• 💔 Build #120131 failed ce675d2
• 💚 Build #119866 succeeded 7133d50
• 💔 Build #119782 failed 78e631a
• 💔 Build #119771 failed 63e8c10

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @lukeelmers

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.