[Fleet] [Meta] Add UI integration tests for Fleet app

[joshdover]

We've had a few recent breakages that likely could have been caught by automated UI integration tests. In #113106, we added support for writing tests in the Cypress test framework which greatly improves the test writing experience and reliability. We should start leveraging this new framework to cover some our most important flows.

We should not use these UI integration tests to cover all edge cases, but should follow the familiar "testing pyramid" model where these tests cover each feature, but not every permutation of inputs and outputs. Unit tests should leveraged for such cases.

These tests will not be truly e2e (due to no running Fleet Server and Agents), but should assert against the expected state in the Fleet API, Elasticsearch APIs, and/or the .fleet-* indices.

This issue is tangential to #115788 which covers the Integrations list behavior.

Areas needing tests

Roughly in priority of importance, we should add tests for the following UI flows:

Onboarding and setup

Preview Give feedback

1. Fleet Server hosts settings - Link
2. Agent binary download source - Link
3. Fleet Server setup (currently skipped) - Link
4. Output settings Link
5. Proxy settings
6. [Fleet] Add cypress tests for Editor and Viewer role under new privileges #125836
   Team:Fleet technical debt +2
   

Agent policies

Preview Give feedback

1. Edit agent policy settings - Link
2. Create an agent policy (partially covered by skipped test)
3. Add an integration to a policy - Partially covered by skipped test

Integration policies

Preview Give feedback

1. Add an integration policy from the Integrations app - Link
2. Edit an integration policy - Link
3. Upgrade an integration policy - Link
4. Create an agent policy from integration policy editor

Integrations

Preview Give feedback

1. Install assets - Link
2. Install integrations - Integrations_real - Integrations_mock
3. Uninstall
4. Reinstall
5. Show assets
6. [Fleet] Add UI integration tests for Integrations app #115788
   Feature:Unified Integrations Team:Fleet chore technical debt tests_needed +5

Agent list view

Preview Give feedback

1. Filtering and search on most common fields - Link
2. [Fleet] Add Cypress tests for agents table #134085
   Team:Fleet +1
   
3. Unenrolling an agent
4. Assigning an agent to a new policy
5. Upgrading an agent (partially covered by bulk actions test
6. Add agent flyout - Link
7. Agent activity flyout
8. Add Fleet server flyout

Agent details view

Preview Give feedback

1. Basic information is displayed
2. Logging view
3. Unenrolling an agent
4. Assigning an agent to a new policy
5. Upgrading an agent

Enrollment tokens view

Preview Give feedback

1. Deleting - Link
2. Creating - Link
3. Filtering and search

Data streams view

Preview Give feedback

1. Filtering and search
2. Reload

Recommendation:

• Keep tests with real API at a minimum, as they are much slower
• Real API should used for high value scenarios only (high impact if broken, most used)
• Mock API can be used for less used, less impactful paths - the idea is that API tests should cover that the backend works as expected too
• Avoid testing the same feature twice, one feature may cover another feature implicitly (e.g. add integration creates agent policy by default)
• e2e tests can have more steps in one testcase, not efficient to separate them to small testcases with one assert (setup takes more time)
• clean up after running the tests, to make them repeatable (e.g. delete created assets)

See as an example the time difference: real API takes minutes, mock API seconds

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[joshdover]

@mostlyjason Would like some feedback on the priority here from a customer perspective. I think we will eventually cover all of this, but it would be good to prioritize the most important flows for customers + areas we expect to making changes in the near future.

[mostlyjason]

Looks like a good order

[juliaElastic]

One thing that I think Chaulet raised earlier, that we should point epr to a mock repo/API in cypress tests, to avoid breakages when epr packages change. I think we could try doing that with this issue. If it needs more effort, it can be raised as a new issue.

[joshdover]

One thing that I think Chaulet raised earlier, that we should point epr to a mock repo/API in cypress tests, to avoid breakages when epr packages change.

I think we should discuss this. Is the benefit of doing this over running the local EPR registry with stable test packages that we can control when they change, worth the downsides? I worry that implementing a mock API will be fragile and could give us a false sense of confidence in the tests.

Additionally, we are going to need to handle package changes, especially for these default packages which are slated to be bundled directly in Kibana as part of #112095. For those, I think we should explicitly test against package changes when they get updated in the Kibana repo.

[criamico]

I'm not sure if the test covering the "add agent" flow is currently tracked in this issue: https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/cypress/e2e/fleet_agent_flyout.cy.ts