Failing test: X-Pack EPM API Integration Tests.x-pack/test/fleet_api_integration/apis/epm/list·ts - EPM Endpoints EPM - list list api tests lists all limited packages from the registry

[kibanamachine]

A test failed on a tracked branch

Error: expected [] to sort of equal [ 'endpoint' ]
    at Assertion.assert (expect.js:100:11)
    at Assertion.eql (expect.js:244:8)
    at Context.<anonymous> (list.ts:58:46)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:73:16) {
  actual: '[]',
  expected: '[\n  "endpoint"\n]',
  showDiff: true
}

First failure: CI Build - main

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jlind23]

@kpollich I might be wrong but isn't this an Endpoint related test?

[kpollich]

I don't think this is actually related to endpoint, but the assertion is expecting the endpoint package to be installed.

kibana/x-pack/test/fleet_api_integration/apis/epm/list.ts

Lines 48 to 59 in 329e845

	it('lists all limited packages from the registry', async function () {
	const fetchLimitedPackageList = async () => {
	const response = await supertest
	.get('/api/fleet/epm/packages/limited')
	.set('kbn-xsrf', 'xxx')
	.expect(200);
	return response.body;
	};
	const listResponse = await fetchLimitedPackageList();
	expect(listResponse.items.sort()).to.eql(['endpoint'].sort());
	});

[juliaElastic]

I'm seeing this error in the logs (happening in a 8.10 pr too):

[00:00:15]         │ proc [kibana] [2023-09-26T13:55:10.808+00:00][WARN ][plugins.fleet] Failure to install package [endpoint]: [ResponseError: resource_not_found_exception
[00:00:15]         │ proc [kibana] 	Root causes:
[00:00:15]         │ proc [kibana] 		resource_not_found_exception: Transform with id [endpoint.metadata_united-default-8.10.2] could not be found]
[00:00:15]         │ proc [kibana] [2023-09-26T13:55:10.809+00:00][ERROR][plugins.fleet] uninstalling endpoint-8.10.2 after error installing: [ResponseError: resource_not_found_exception
[00:00:15]         │ proc [kibana] 	Root causes:
[00:00:15]         │ proc [kibana] 		resource_not_found_exception: Transform with id [endpoint.metadata_united-default-8.10.2] could not be found]
[00:00:15]         │ proc [kibana] [2023-09-26T13:55:10.852+00:00][DEBUG][plugins.fleet] Created transform: endpoint.metadata_current-default-8.10.2

Seems flaky though, can't reproduce locally.

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[jbudz]

/skip

[kibanamachine]

Skipped

main: e70181a

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[mistic]

Skipped.

main: df569d0
8.11: 886ac2c

[kpollich]

Tests don't seem to find the bundled packages here, not sure if it's relevant:

proc [kibana] [2023-11-05T05:36:37.859+00:00][WARN ][plugins.fleet] Bundled package directory /tmp/fleet_bundled_packages does not exist. All packages will be sourced from http://localhost:6104. {"service":{"node":{"roles":["background_tasks","ui"]}}}

The failure in question is expecting endpoint to be installed, but it isn't

1)    EPM Endpoints
--
  | EPM - list
  | list api tests
  | lists all limited packages from the registry:
  |  
  |  
  | Error: expected [] to sort of equal [ 'endpoint' ]
  | + expected - actual
  |  
  | -[]
  | +[
  | +  "endpoint"
  | +]

[kibanamachine]

New failure: CI Build - 8.11

[kibanamachine]

New failure: CI Build - 8.11

[kibanamachine]

New failure: CI Build - 8.11

[juliaElastic]

Tests don't seem to find the bundled packages here, not sure if it's relevant

@kpollich It is relevant, as the test fails on endpoint not being in registry, similar to this one: #163203 (comment)

Found that the bundled packages are downloaded in ci build, but not in the expected /tmp/fleet_bundled_packages folder, maybe this changed in ci some time back:
https://buildkite.com/elastic/kibana-pull-request/builds/186974#018cf878-6ed6-45b1-b553-31a5a0c51071

│ info Fetching fleet packages from package registry
--
...
  | │ debg Downloaded 1616079 bytes to /dev/shm/builds/kb-n2-16-spot-aa7d3da422b2f059/elastic/kibana-pull-request/kibana/build/kibana/x-pack/plugins/fleet/target/bundled_packages/endpoint-8.11.1.zip
  | │ debg Downloaded https://epr.elastic.co/epr/endpoint/endpoint-8.11.1.zip
 ...
  | │ info Copied @kbn/fleet-plugin into build

[kpollich]

Interesting. The closest thing I found to the introduction of the /tmp path is this PR: #126915, but I don't see where we ever would've told this script to download to /tmp. Yet it was definitely working at some point 🤔

[juliaElastic]

Interesting. The closest thing I found to the introduction of the /tmp path is this PR: #126915, but I don't see where we ever would've told this script to download to /tmp. Yet it was definitely working at some point 🤔

Thanks for the link!
From what I understand, we never downloaded the bundled packages to the /tmp folder, but used the code here to copy from fixtures to the tmp folder to test a bundled package (nginx). So the other tests always tried to download the endpoint package from the registry I guess, and sometimes don't find it. I'll try to fix this to add the endpoint package to the bundled fixtures, as it is supposed to be a bundled package.