Fleet packages/assets causing failures in savedObjects management FTR tests

[pgayvallet]

I'm opening this issue to discuss about the interactions of fleet assets and our SOM (savedObjects management) tests. The introduction of savedObjects automatically installed at startup by fleet is causing a lot of pain in our SOM FTR tests.

For example, in #116058, the test fails because it expect a fresh install of Kibana to only contains one object (the config object), which is no longer the case because of fleet assets. In https://github.com/elastic/kibana/pull/121498/files, I tried to work around the fleet installed SOs by installing the empty_kibana archive before the tests, however even with this, the failures are still present on CI, so not even sure when exactly these assets got installed or if they get reinstalled somehow if removed by an external actor/script such as the esArchiver.

We also have a lot of other saved objects management tests suites that are currently disabled because they were considered 'flaky' due to the introduction of fleet packages. For example reenabling #118272 is not possible because it seems to manipulate objects from the SOM table. However due to the fleet assets, our tests objects are spread into multiple pages of the table, causing any bulk operation to not be possible (even if we were to adapt the page to seek the page containing our test objects, they wouldn't all be present on this page)

So, in short, to re-enable most of the skipped SOM FTR tests, we'll need a way to reliability get to a state where we know for sure that the SO objects presents are only the ones from the datasets.

@elastic/fleet josh linked me #116591, but I'm not sure that this would even solves our problem here, because it would not disable fleet assets, just use a local repo for it, right?

cc @joshdover

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[joshdover]

@elastic/fleet josh linked me #116591, but I'm not sure that this would even solves our problem here, because it would not disable fleet assets, just use a local repo for it, right?

No that doesn't solve our problem here but it would help with the situation where the number of objects changes due to the Fleet package that is being installed changing.

I think the best option is to just skip these tests until #108456 is implemented which will stop installing these Saved Objects on Kibana boot. This is scheduled for the next release and I expect work on it to begin next week. We could come up with workarounds (like disabling Fleet setup), but I think it's actually riskier and much more effort to implement those and track removing them before the next release rather than waiting a week or two longer for the proper fix.

Does that work for you? Is there planned work in this area that we need these tests enabled for?

[pgayvallet]

Does that work for you? Is there planned work in this area that we need these tests enabled for?

Yea it definitely would, we're not in a hurry to re-enable those tests. If we know that short/mid term these issues will go away by removing the default integrations, it's perfectly fine with us

[joshdover]

Great, I'll ping back here once this should be resolved. Assigning myself for now so I don't forget.

[joshdover]

@pgayvallet Moving this back to you, we've stopped installing these by default on the 8.1 and main branches.

[pgayvallet]

Thanks, gonna close this, to unblock the associated failing test issues