Failing ES Promotion: X-Pack Endpoint Functional Tests.x-pack/test/security_solution_endpoint/apps/endpoint - endpoint "before all" hook in "endpoint"

[mistic]

X-Pack Endpoint Functional Tests
x-pack/test/security_solution_endpoint/apps/endpoint

endpoint "before all" hook in "endpoint"

This failure is preventing the promotion of the current Elasticsearch nightly snapshot.

For more information on the Elasticsearch snapshot promotion process including how to reproduce using the unverified nightly ES build: https://www.elastic.co/guide/en/kibana/master/development-es-snapshots.html

• Failed promotion job
• Test Failure

EndpointDataLoadingError: security_exception
	Root causes:
		security_exception: Cannot create transform [endpoint.metadata_united-default-8.8.0-preview-2] because user kibana_system lacks the required permissions [.fleet-agents*:[read, view_index_metadata], .metrics-endpoint.metadata_united_default:[], metrics-endpoint.metadata_current_default*:[]]
    at installOrUpgradeEndpointFleetPackage (setup_fleet_for_endpoint.ts:119:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Context.<anonymous> (index.ts:35:7)
    at Object.apply (wrap_function.js:73:16) {
  meta: [
    {
      name: 'endpoint',
      statusCode: 500,
      error: 'security_exception\n' +
        '\tRoot causes:\n' +
        '\t\tsecurity_exception: Cannot create transform [endpoint.metadata_united-default-8.8.0-preview-2] because user kibana_system lacks the required permissions [.fleet-agents*:[read, view_index_metadata], .metrics-endpoint.metadata_united_default:[], metrics-endpoint.metadata_current_default*:[]]'
    }
  ]
}

[elasticmachine]

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

[mistic]

Skipped.

main: c39031e

[kevinlog]

I reproduced this permissions issue in transform installation easily by using the latest ES from source.

The difference seems to be the latest ES code. If I switch between source and the older snapshot, it will work on the older snapshot.

Investigating further

[kevinlog]

I think it may be related to this change: elastic/elasticsearch#94724

[kevinlog]

I confirmed with @qn895 that elastic/elasticsearch#94724 changed the way that we validate permissions in transforms.

@qn895 is working on this Kibana PR (#154665) which is meant to go with the ES PR referenced above.

After the permissions issue is resolved, the functionality and tests should be restored.

cc @przemekwitek @droberts195

[kevinlog]

Additional skips here: #154813

[kevinlog]

Tests should be able to be unskipped after we get another ES snapshot promotion including this change: elastic/elasticsearch#95187

[kevinlog]

closing with #154813