Surface deprecation logs from Kibana CI

[rudolf]

To help teams audit, fix and suppress deprecation warnings originating from Elastic products we need to instrument Kibana CI.

Short term:

• Surface deprecation warnings in CI logs
• Provide the necessary context to make it easy for teams to action
  1. Which test triggered the deprecation log
  2. The complete query that caused the deprecation log
  3. A stack trace (the nature of Nodejs means stack traces aren't always useful in 8.x. Due to the architecture of the elasticsearch-js client 7.x stack traces are never useful)

Medium term: @elastic/kibana-operations
Create a workflow/process to manage deprecation logs in Kibana

1. PR's fail if they introduce new deprecation logs. These can be ignored by adding them to an ignored_deprecations file which uses the deprecation message and test name as unique keys.
2. If ES promotion fails due to new deprecation logs, the operations team will add the new logs to the ignored_deprecations file and create an issue for the respective team to address.

We should distinguish between log originating from user supplied queries (e.g. console), kibana or originating from test suites themselves.

There are several ways we could surface deprecation logs:

1. Deprecation logger to stdout

By setting the following log4j2.properties we can direct deprecation logs in JSON format to stdout. The ToolingLog could then write these to file so that they appear in the final CI logs.

appender.deprecation_console.type = Console
appender.deprecation_console.name = deprecation_console
appender.deprecation_console.layout.type = ECSJsonLayout
appender.deprecation_console.layout.dataset = deprecation.elasticsearch

logger.deprecation.name = org.elasticsearch.deprecation
logger.deprecation.level = WARN
logger.deprecation.appenderRef.deprecation_console.ref = deprecation_console
logger.deprecation.appenderRef.header_warning.ref = header_warning
logger.deprecation.additivity = false

Example log line:

{
    "@timestamp": "2021-11-30T22:42:27.955Z",
    "log.level": "CRITICAL",
    "data_stream.dataset": "deprecation.elasticsearch",
    "data_stream.namespace": "default",
    "data_stream.type": "logs",
    "ecs.version": "1.7",
    "elasticsearch.event.category": "api",
    "event.code": "open_system_index_access",
    "message": "this request accesses system indices: [.kibana_8.0.0_001], but in a future major version, direct access to system indices will be prevented by default",
    "service.name": "ES_ECS",
    "process.thread.name": "elasticsearch[rudolf-elastic-mbp.home][system_critical_read][T#2]",
    "log.logger": "org.elasticsearch.deprecation.cluster.metadata.IndexNameExpressionResolver",
    "event.dataset": "deprecation.elasticsearch",
    "elasticsearch.cluster.uuid": "sod7KxtGQiCckgLNfaQMcA",
    "elasticsearch.node.id": "LBLoRyyaQduqJjvampkubQ",
    "elasticsearch.node.name": "rudolf-elastic-mbp.home",
    "elasticsearch.cluster.name": "elasticsearch"
}

2. warning response header

Each API request that triggers a deprecation will include a warning header in it's response for example:

299 Elasticsearch-8.1.0-SNAPSHOT-fe7c2d59d80ae9f75c513f7d89879acef665ebe0 "this request accesses system indices: [.kibana_task_manager_8.1.0_001], but in a future major version, direct access to system indices will be prevented by default"

Similar to how elasticsearch.query logger context (previously elasticsearch.logQueries config option) works, we can log a lot of extra context like the entire query and the stack trace:

ES DEPRECATION: 299 Elasticsearch-8.1.0-SNAPSHOT-fe7c2d59d80ae9f75c513f7d89879acef665ebe0 "this request accesses system indices: [.kibana_task_manager_8.1.0_001], but in a future major version, direct access to system indices will be prevented by default"
Origin: kibana
Stack trace:
    at ApmClient.DeleteApi [as delete](/Users/rudolf/dev/kibana/node_modules/@elastic/elasticsearch/src/api/api/delete.ts:59:10)
Query:
404
DELETE /.kibana_task_manager_8.1.0/_doc/task%3Adata_enhanced_search_sessions_cleanup?refresh=wait_for

3. Query the .ds-.logs-deprecation.elasticsearch-default

We explored this approach in #117937 but this provides the least context since we can only collect the deprecation logs at a set interval or time, e.g. at the end of a test suite, making it hard to trace back to the offending test and even harder to the corresponding line of source code.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)