[Fleet] cypress setup #113106

juliaElastic · 2021-09-27T12:00:02Z

Summary

Cypress setup for Fleet. ON week work.

Setup for Fleet Server and Agent is from Bryan Clement's pr here: #108759

To run locally:

cd x-pack/plugins/fleet
yarn cypress:open-as-ci

Make sure to stop any agent instances before starting, otherwise you might get this grpc port conflict: elastic/beats#28118

Added cypress to ci run:
https://github.com/elastic/kibana/pull/113106/checks?check_run_id=3744593878

Cypress major upgrade just happened with this: #104301

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

juliaElastic · 2021-09-29T14:58:35Z

@elasticmachine merge upstream

juliaElastic · 2021-10-01T08:18:00Z

@elasticmachine merge upstream

juliaElastic · 2021-10-04T11:56:27Z

@joshdover @jen-huang this is the draft pr that I've done during ON week to add cypress setup to Fleet, do we want to merge it or wait until it is presented to the team?

The caveat is that this task adds 2 minutes to build time, I don't know yet if these test tasks are parallelised on CI.
Mostly the API calls take that long, that is why it is recommended to use stubbed network requests for most cases.

joshdover · 2021-10-04T12:50:30Z

If no one on the team is opposed to moving forward with Cypress, I'm +1 on moving this into review.

juliaElastic · 2021-10-04T12:53:40Z

@elasticmachine merge upstream

juliaElastic · 2021-10-04T13:02:04Z

x-pack/plugins/fleet/cypress/integration/integrations.spec.ts

    navigateTo(INTEGRATIONS);
    cy.get(INTEGRATIONS_CARD).contains(integration).click();
    addIntegration();
    cy.get(INTEGRATION_NAME_LINK).contains('apache-');
+  }
+
+  it('[Mocked requests] should display Apache integration in the Policies list once installed ', () => {


this test is the duplication of the one above, but with mocked requests to showcase network stubbing.
Pros: quicker test, isolation of UI testing
Cons: lot of extra setup needed (fixture files, intercept each api call)
I think for critical testcases real e2e test would be better, mocked requests are useful when the goal is to test complex UI logic without relying on real backend.
See https://docs.cypress.io/guides/guides/network-requests

I think real request will make sense here. But maybe we should do the same things as we do with our API integration tests where we have a docker package registry locally otherwise I think we can have some flakiness here. (This should be done in a following PR already a lot of good stuff here)

elasticmachine · 2021-10-04T13:02:10Z

Pinging @elastic/fleet (Team:Fleet)

juliaElastic · 2021-10-28T17:59:04Z

x-pack/test/fleet_cypress/fleet_server.ts

+          '--env',
+          'FLEET_SERVER_ENABLE=true',
+          '--env',
+          `FLEET_SERVER_ELASTICSEARCH_HOST=http://host.docker.internal:${this.esConfig.port}`,


this was working locally, though it is not on CI, trying to figure out why, not able to connect to ES

2021-10-28T17:56:55.723Z INFO cmd/enroll_cmd.go:760 Fleet Server - Starting -- | 2021-10-28T17:56:56.527Z ERROR status/reporter.go:236 Elastic Agent status changed to: 'error' | 2021-10-28T17:56:56.528Z ERROR log/reporter.go:36 2021-10-28T17:56:56Z - message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to FAILED: Error - dial tcp 172.17.0.1:6102: connect: connection refused - type: 'ERROR' - sub_type: 'FAILED' | 2021-10-28T17:56:57.726Z INFO cmd/enroll_cmd.go:760 Fleet Server - Error - dial tcp 172.17.0.1:6102: connect: connection refused

juliaElastic · 2021-11-01T07:43:27Z

@elasticmachine merge upstream

juliaElastic · 2021-11-01T12:53:47Z

x-pack/test/fleet_cypress/config.ts

+        // define custom es server here
+        // API Keys is enabled at the top level
+        'xpack.security.enabled=true',
+        'http.host=0.0.0.0',


finally, fleet_server with docker works by adding http.host to make es accessible from the docker container

joshdover

This is great, tested locally and was able to follow the flow of how things are linked together pretty well. I see no reason to hold on merging this and iterating as we go. Thank you for doing this, it's going to be very helpful for us going forward.

joshdover · 2021-10-18T14:25:52Z

.gitignore

+elastic-agent-*
+fleet-server-*
+elastic-agent.yml
+fleet-server.yml


nit: new line

joshdover · 2021-11-01T16:54:47Z

x-pack/plugins/fleet/cypress/support/commands.js

+// commands please read more here:
+// https://on.cypress.io/custom-commands
+// ***********************************************
+//


nit: can we make this a ts file?

joshdover · 2021-11-01T16:55:18Z

x-pack/test/fleet_cypress/agent.ts

+    // this.log.info('Updating the default agent output');
+    // const {
+    //   data: {
+    //     items: [defaultOutput],
+    //   },
+    // } = await axios.get(this.params.kibanaUrl + '/api/fleet/outputs', this.requestOptions);
+
+    // await axios.put(
+    //   `${this.params.kibanaUrl}/api/fleet/outputs/${defaultOutput.id}`,
+    //   { hosts: [this.params.esHost] },
+    //   this.requestOptions
+    // );


nit: remove

joshdover · 2021-11-01T16:57:06Z

x-pack/plugins/fleet/cypress/plugins/index.js

+module.exports = (_on, _config) => {
+  // `on` is used to hook into various events Cypress emits
+  // `config` is the resolved Cypress config
+};


can we make this a ts file?

juliaElastic · 2021-11-02T12:18:04Z

@elasticmachine merge upstream

kibanamachine · 2021-11-02T14:13:25Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	615.3KB	615.5KB	+177.0B

History

💔 Build #3527 failed c29d688
💔 Build #3504 failed c29d688
💚 Build #3167 succeeded db35a93
💔 Build #3134 failed 2a65c57
💔 Build #3122 failed ebb8f64

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* added cypress setup for fleet * updated readme * fixed types * fix unused * starting up fleet server in cypress * added more fleet tests * added package update test * updated readme * trying jenkins run * trying jenkins run * trying jenkins run * fix types, cypress config * example of mocked requests * added getBySel * fix test after refactor * removed duplication * fix tests * added to buildkite * updated tests * starting agent with docker * trying to fix ip address * ifconfig * ip address * ip address * ip address * type fix * remove extra logging * fixed test * fix buildkite * cleanup * cleanup * using docker for fleet_server * fix * fix * trying to fix * update config * test * test * test * test * revert changes * test * test * static ip on linux * docker version * try again * fixed review comments * fixed types Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Patryk Kopycinski <[email protected]>

kibanamachine · 2021-11-02T14:18:21Z

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.

* [Fleet] cypress setup (#113106) * added cypress setup for fleet * updated readme * fixed types * fix unused * starting up fleet server in cypress * added more fleet tests * added package update test * updated readme * trying jenkins run * trying jenkins run * trying jenkins run * fix types, cypress config * example of mocked requests * added getBySel * fix test after refactor * removed duplication * fix tests * added to buildkite * updated tests * starting agent with docker * trying to fix ip address * ifconfig * ip address * ip address * ip address * type fix * remove extra logging * fixed test * fix buildkite * cleanup * cleanup * using docker for fleet_server * fix * fix * trying to fix * update config * test * test * test * test * revert changes * test * test * static ip on linux * docker version * try again * fixed review comments * fixed types Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Patryk Kopycinski <[email protected]> * using 8.0.0 docker image for backport Co-authored-by: Julia Bardi <[email protected]> Co-authored-by: Patryk Kopycinski <[email protected]> Co-authored-by: Julia Bardi <[email protected]>

juliaElastic and others added 13 commits September 27, 2021 13:56

added cypress setup for fleet

3cb4113

Merge branch 'elastic:master' into fleet-cypress

d74478d

updated readme

648bf47

fixed types

7099dc1

fix unused

bb21e5c

starting up fleet server in cypress

98164cf

added more fleet tests

5f28762

added package update test

4261fde

updated readme

72fa3f7

trying jenkins run

f47cafd

trying jenkins run

6c557ef

trying jenkins run

ec6714e

fix types, cypress config

9411f5d

kibanamachine and others added 3 commits September 29, 2021 10:59

Merge branch 'master' into fleet-cypress

f1e8984

example of mocked requests

15ddd5c

Merge branch 'master' into fleet-cypress

8fa29b3

kibanamachine and others added 2 commits October 1, 2021 04:18

Merge branch 'master' into fleet-cypress

9464634

added getBySel

a3983bd

Merge branch 'master' into fleet-cypress

f285b11

juliaElastic marked this pull request as ready for review October 4, 2021 12:54

juliaElastic requested review from a team as code owners October 4, 2021 12:54

juliaElastic commented Oct 4, 2021

View reviewed changes

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Oct 4, 2021

trying to fix

e550286

juliaElastic commented Oct 28, 2021

View reviewed changes

patrykkopycinski added 9 commits October 29, 2021 10:24

Merge branch 'master' of github.com:elastic/kibana into fleet-cypress

bd01ccb

update config

4a5d233

test

9d63e66

test

c22af32

test

cd83d72

test

66e87ba

revert changes

10d36bf

test

722994a

test

3faaa90

kibanamachine and others added 4 commits November 1, 2021 03:43

Merge branch 'main' into fleet-cypress

6bf2915

static ip on linux

ebb8f64

docker version

2a65c57

try again

db35a93

juliaElastic commented Nov 1, 2021

View reviewed changes

joshdover approved these changes Nov 1, 2021

View reviewed changes

juliaElastic added 2 commits November 2, 2021 09:26

fixed review comments

c29d688

fixed types

7819abc

Merge branch 'main' into fleet-cypress

ce6c85b

juliaElastic merged commit 157a371 into elastic:main Nov 2, 2021

kibanamachine mentioned this pull request Nov 2, 2021

[8.0] [Fleet] cypress setup (#113106) #117138

Merged

patrykkopycinski mentioned this pull request Dec 3, 2021

[Osquery] Add Osquery cypress tests to buildkite #115902

Merged

joshdover mentioned this pull request Dec 16, 2021

[Fleet] [Meta] Add UI integration tests for Fleet app #121436

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] cypress setup #113106

[Fleet] cypress setup #113106

juliaElastic commented Sep 27, 2021 •

edited

Loading

juliaElastic commented Sep 29, 2021 •

edited

Loading

juliaElastic commented Oct 1, 2021

juliaElastic commented Oct 4, 2021 •

edited

Loading

joshdover commented Oct 4, 2021

juliaElastic commented Oct 4, 2021 •

edited

Loading

juliaElastic Oct 4, 2021

nchaulet Oct 12, 2021

elasticmachine commented Oct 4, 2021

juliaElastic Oct 28, 2021 •

edited

Loading

juliaElastic commented Nov 1, 2021

juliaElastic Nov 1, 2021

joshdover left a comment

joshdover Oct 18, 2021

joshdover Nov 1, 2021

joshdover Nov 1, 2021

joshdover Nov 1, 2021

juliaElastic commented Nov 2, 2021

kibanamachine commented Nov 2, 2021

kibanamachine commented Nov 2, 2021

[Fleet] cypress setup #113106

[Fleet] cypress setup #113106

Conversation

juliaElastic commented Sep 27, 2021 • edited Loading

Summary

Checklist

Risk Matrix

For maintainers

juliaElastic commented Sep 29, 2021 • edited Loading

juliaElastic commented Oct 1, 2021

juliaElastic commented Oct 4, 2021 • edited Loading

joshdover commented Oct 4, 2021

juliaElastic commented Oct 4, 2021 • edited Loading

juliaElastic Oct 4, 2021

Choose a reason for hiding this comment

nchaulet Oct 12, 2021

Choose a reason for hiding this comment

elasticmachine commented Oct 4, 2021

juliaElastic Oct 28, 2021 • edited Loading

Choose a reason for hiding this comment

juliaElastic commented Nov 1, 2021

juliaElastic Nov 1, 2021

Choose a reason for hiding this comment

joshdover left a comment

Choose a reason for hiding this comment

joshdover Oct 18, 2021

Choose a reason for hiding this comment

joshdover Nov 1, 2021

Choose a reason for hiding this comment

joshdover Nov 1, 2021

Choose a reason for hiding this comment

joshdover Nov 1, 2021

Choose a reason for hiding this comment

juliaElastic commented Nov 2, 2021

kibanamachine commented Nov 2, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

History

kibanamachine commented Nov 2, 2021

💚 Backport successful

juliaElastic commented Sep 27, 2021 •

edited

Loading

juliaElastic commented Sep 29, 2021 •

edited

Loading

juliaElastic commented Oct 4, 2021 •

edited

Loading

juliaElastic commented Oct 4, 2021 •

edited

Loading

juliaElastic Oct 28, 2021 •

edited

Loading