[Fleet] Test Elasticsearch deprecation logs generated by Fleet

[joshdover]

In order to improve the upgrade experience for 8.0, we should be sure Fleet's usage of Elasticsearch APIs do not generate any deprecation logs that could confuse the customer when trying to find deprecation usages in their own tooling. I don't currently expect Fleet to be an offender here since the code base is very new, but we check anyways.

This should be done on the 7.last release and should combine both automated and manual tests:

• Manual testing
  • Go through the main package lifecycle install / upgrade / remove process
  • Perform agent actions
  • Agent logs UI
• Automated testing
  • Run the fleet_api_integration FTR suite
    • You'll want to do this in 'split' mode so you can inspect the logs stored in Elasticsearch after running the suite. Run node scripts/functional_test_runner.js --config x-pack/test/fleet_api_integration/config.ts in one shell and after ES and Kibana have fully started, run follow the instructions in another shell to run the tests.
  • https://github.com/elastic/e2e-testing ?

After running any of these tests, we should query and log out all documents from the .logs-deprecation.elasticsearch-default data stream to and try to track down any Fleet-specific records. Example command (requires curl and jq):

curl -u elastic:changeme "localhost:9200/.logs-deprecation.elasticsearch-default/_search?size=10000" | jq '.hits.hits | map(._source)'

As I mentioned above, I don't expect Fleet to be an offender. But if needed, we may be able to use the x-opaque-id header feature that propogates to the deprecation logs (see docs below) somehow if we suspect we're adding anything. @rudolf any guidance on a way to get our plugin ID in to the x-opaque-id header we send to Elasticsearch?

Helpful links:

• Elasticsearch docs on deprecation logging
• Elasticsearch REST API breaking changes in 8.0

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jen-huang]

@joshdover Thanks for filing this. Should this be on the iteration board or backlog?

[rudolf]

@joshdover I've started with #120044 which I think is the approach that will provide the richest context for Kibana developers about the origin of a deprecation.

I hope to be able to streamline the whole process in the next week or so #120043

But if you want to be proactive you can find the fleet_api_integration test suites in the build output from #117937 If at the bottom of the fleet_api_integration output there aren't any deprecation logs you're good to go (similar to your instructions just don't have to run it locally)

[joshdover]

Thanks for filing this. Should this be on the iteration board or backlog?

Wasn't sure when you put things into the Iteration board so I just put into Backlog for now since it wasn't an 8.0 FF item. Should we just go ahead and move it to Iteration?

EDIT: nvm Julia already moved it :)

[juliaElastic]

I think it makes sense to add since we are working on it this iteration.

[juliaElastic]

@joshdover This is what I found so far.
I see only one warning clearly related to fleet. Are we planning to rename fleet indices starting with dot or should we suppress this warning? I think these are considered system indices, right?

For the rest Rudolf's pr #120044 would help to track down the stacktrace.

Fleet Integration tests

"index name [.fleet-agent-7] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices"
"this request accesses system indices: [.security-7], but in a future major version, direct access to system indices will be prevented by default",

Manual testing

"index name [.siem-signals-default-000001] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices",
"index name [.metrics-endpoint.metadata_united_default] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices",
"this request accesses system indices: [.transform-internal-007], but in a future major version, direct access to system indices will be prevented by default",
"[types removal] Using include_type_name in create index requests is deprecated. The parameter will be removed in the next major version.",
"Legacy index templates are deprecated in favor of composable templates.",
"The user [kibana] is deprecated and will be removed in a future version of Elasticsearch. Please use the [kibana_system] user instead."

Deprecation logs from CI are here: #117937 , no specific fleet errors here either.

Note: I couldn't fully run fleet_integration_tests locally, it starts breaking after some point with this error in ES logs:

│ proc [kibana] Unhandled Promise rejection detected:
   │ proc [kibana] 
   │ proc [kibana] ResponseError: Saved object index alias [.kibana_7.16.0] not found: index_not_found_exception: [index_not_found_exception] Reason: no such index [.kibana_7.16.0] and [require_alias] request flag is [true] and [.kibana_7.16.0] is not an alias
   │ proc [kibana]     at onBody (/Users/juliabardi/kibana/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:367:23)
   │ proc [kibana]     at IncomingMessage.onEnd (/Users/juliabardi/kibana/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:291:11)
   │ proc [kibana]     at IncomingMessage.emit (node:events:402:35)
   │ proc [kibana]     at endReadableNT (node:internal/streams/readable:1343:12)
   │ proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
   │ proc [kibana]   meta: {
   │ proc [kibana]     body: { error: [Object], status: 404 },

Test output:

ResponseError: index_not_found_exception: [index_not_found_exception] Reason: no such index [logs-log.log-test]                                                                                                                                                

[joshdover]

I see only one warning clearly related to fleet. Are we planning to rename fleet indices starting with dot or should we suppress this warning? I think these are considered system indices, right?

"index name [.fleet-agent-7] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices"

This is curious because Kibana already provides the x-elastic-product-origin: kibana header on all requests to Elasticsearch which should allow us to access these indices without a warning. I also see that Fleet's system index configuration allows the kibana product header here for the .fleet-agents index: https://github.com/elastic/elasticsearch/blob/86978e87b877b304d81ae749266c248607da6f7e/x-pack/plugin/fleet/src/main/java/org/elasticsearch/xpack/fleet/Fleet.java#L159

Note that this warning says the error in on the .fleet-agent-7 index not the .fleet-agents-7 index (notice the s). I can't find any code in Kibana that references this index. @nchaulet any ideas?

[joshdover]

"index name [.metrics-endpoint.metadata_united_default] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices",

Curious where this one is coming from. @juliaElastic does this happen when you install or upgrade the endpoint package? Wonder if we need to adjust any of our settings update or mapping update code to have the allow_restricted_indices query option?

[juliaElastic]

@joshdover yes, I tested with endpoint security package. I see that index mentioned in security_solution plugin here.
Shouldn't we notify Security to update it or is it Fleet's task?

[joshdover]

Shouldn't we notify Security to update it or is it Fleet's task?

If it's not from package install, upgrade, or removal then I think we can let Security handle it.

[nchaulet]

@joshdover I found the issue with the bad named fleet agent it's one of our fixture that contains a typo #120098

[juliaElastic]

I've ran e2e-tests and didn't see any new deprecation logs (though some tests failed for me locally).

[joshdover]

@juliaElastic I think we can close this now unless there's any additional testing you think we need.

[juliaElastic]

@joshdover
I checked the stacktraces locally based on Rudolf's pr. Unfortunately they are not really meaningful. At least in queries I
didn't see anything specific to fleet.

   │ proc [kibana] server    log   [14:39:54.331] [error][data][deprecation][elasticsearch] ES DEPRECATION: 299 Elasticsearch-7.16.0-SNAPSHOT-04b1b20587394c0581beab262fd89e582126af65 "The user [kibana] is deprecated and will be removed in a future version of Elasticsearch. Please use the [kibana_system] user instead."
   │ proc [kibana] Origin:false
   │ proc [kibana] Stack trace:
   │ proc [kibana]     at IncomingMessage.emit (node:events:402:35)
   │ proc [kibana]     at endReadableNT (node:internal/streams/readable:1343:12)
   │ proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:83:21)
   │ proc [kibana] Query:
   │ proc [kibana] 200
   │ proc [kibana] GET /

   │ proc [kibana] server    log   [14:40:16.096] [error][data][deprecation][elasticsearch] ES DEPRECATION: 299 Elasticsearch-7.16.0-SNAPSHOT-04b1b20587394c0581beab262fd89e582126af65 "Legacy index templates are deprecated in favor of composable templates."
   │ proc [kibana] Origin:false
   │ proc [kibana] Stack trace:
   │ proc [kibana]     at IncomingMessage.emit (node:events:402:35)
   │ proc [kibana]     at endReadableNT (node:internal/streams/readable:1343:12)
   │ proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:83:21)
   │ proc [kibana] Query:
   │ proc [kibana] 200
   │ proc [kibana] PUT /_template/.kibana_security_session_index_template_1
   │ proc [kibana] {"index_patterns":[".kibana_security_session_1"],"order":1000,"settings":{"index":{"number_of_shards":1,"number_of_replicas":0,"auto_expand_replicas":"0-1","priority":1000,"refresh_interval":"1s","hidden":true}},"mappings":{"dynamic":"strict","properties":{"usernameHash":{"type":"keyword"},"provider":{"properties":{"name":{"type":"keyword"},"type":{"type":"keyword"}}},"idleTimeoutExpiration":{"type":"date"},"lifespanExpiration":{"type":"date"},"accessAgreementAcknowledged":{"type":"boolean"},"content":{"type":"binary"}}}}
   │ info [o.e.c.m.MetadataIndexTemplateService] [node-01] adding component template [.alerts-observability.metrics.alerts-mappings]

   │ proc [kibana] server    log   [14:47:47.516] [error][data][deprecation][elasticsearch] ES DEPRECATION: 299 Elasticsearch-7.16.0-SNAPSHOT-04b1b20587394c0581beab262fd89e582126af65 "index name [.metrics-endpoint.metadata_united_default] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices"
   │ proc [kibana] Origin:false
   │ proc [kibana] Stack trace:
   │ proc [kibana]     at IncomingMessage.emit (node:events:402:35)
   │ proc [kibana]     at endReadableNT (node:internal/streams/readable:1343:12)
   │ proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:83:21)
   │ proc [kibana] Query:
   │ proc [kibana] 200
   │ proc [kibana] POST /_transform/endpoint.metadata_united-default-1.2.2/_start

   │ proc [kibana] server    log   [14:58:06.045] [error][data][deprecation][elasticsearch] ES DEPRECATION: 299 Elasticsearch-7.16.0-SNAPSHOT-04b1b20587394c0581beab262fd89e582126af65 "this request accesses system indices: [.security-7, .transform-internal-007], but in a future major version, direct access to system indices will be prevented by default"
   │ proc [kibana] Origin:false
   │ proc [kibana] Stack trace:
   │ proc [kibana]     at IncomingMessage.emit (node:events:402:35)
   │ proc [kibana]     at endReadableNT (node:internal/streams/readable:1343:12)
   │ proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:83:21)
   │ proc [kibana] Query:
   │ proc [kibana] 200
   │ proc [kibana] GET /_all/_rollup/data

  │ proc [kibana] server    log   [14:59:53.784] [error][data][deprecation][elasticsearch] ES DEPRECATION: 299 Elasticsearch-7.16.0-SNAPSHOT-04b1b20587394c0581beab262fd89e582126af65 "The user [kibana] is deprecated and will be removed in a future version of Elasticsearch. Please use the [kibana_system] user instead."
   │ proc [kibana] Origin:false
   │ proc [kibana] Stack trace:
   │ proc [kibana]     at IncomingMessage.emit (node:events:402:35)
   │ proc [kibana]     at endReadableNT (node:internal/streams/readable:1343:12)
   │ proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:83:21)
   │ proc [kibana] Query:
   │ proc [kibana] 200
   │ proc [kibana] GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip

[joshdover]

Ok let's just close and if anything else comes up we'll address it. These warnings are going to be suppressed by Kibana anyways so it's not a release blocker.