[Fleet] cypress setup (elastic#113106)

* added cypress setup for fleet * updated readme * fixed types * fix unused * starting up fleet server in cypress * added more fleet tests * added package update test * updated readme * trying jenkins run * trying jenkins run * trying jenkins run * fix types, cypress config * example of mocked requests * added getBySel * fix test after refactor * removed duplication * fix tests * added to buildkite * updated tests * starting agent with docker * trying to fix ip address * ifconfig * ip address * ip address * ip address * type fix * remove extra logging * fixed test * fix buildkite * cleanup * cleanup * using docker for fleet_server * fix * fix * trying to fix * update config * test * test * test * test * revert changes * test * test * static ip on linux * docker version * try again * fixed review comments * fixed types Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Patryk Kopycinski <[email protected]>
kibanamachine · Nov 2, 2021 · 01297be · 01297be
1 parent 7f6a5fc
commit 01297be
Show file tree

Hide file tree

Showing 42 changed files with 4,248 additions and 4 deletions.
diff --git a/.buildkite/pipelines/pull_request/fleet_cypress.yml b/.buildkite/pipelines/pull_request/fleet_cypress.yml
@@ -0,0 +1,11 @@
+steps:
+  - command: .buildkite/scripts/steps/functional/fleet_cypress.sh
+    label: 'Fleet Cypress Tests'
+    agents:
+      queue: ci-group-6
+    depends_on: build
+    timeout_in_minutes: 120
+    retry:
+      automatic:
+        - exit_status: '*'
+          limit: 1
diff --git a/.buildkite/scripts/pipelines/pull_request/pipeline.js b/.buildkite/scripts/pipelines/pull_request/pipeline.js
@@ -76,6 +76,16 @@ const uploadPipeline = (pipelineContent) => {
       pipeline.push(getPipeline('.buildkite/pipelines/pull_request/apm_cypress.yml'));
     }
 
+    if (
+      (await doAnyChangesMatch([
+        /^x-pack\/plugins\/fleet/,
+        /^x-pack\/test\/fleet_cypress/,
+      ])) ||
+      process.env.GITHUB_PR_LABELS.includes('ci:all-cypress-suites')
+    ) {
+      pipeline.push(getPipeline('.buildkite/pipelines/pull_request/fleet_cypress.yml'));
+    }
+
     if (await doAnyChangesMatch([/^x-pack\/plugins\/uptime/])) {
       pipeline.push(getPipeline('.buildkite/pipelines/pull_request/uptime.yml'));
     }

diff --git a/.buildkite/scripts/steps/functional/fleet_cypress.sh b/.buildkite/scripts/steps/functional/fleet_cypress.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+source .buildkite/scripts/common/util.sh
+
+.buildkite/scripts/bootstrap.sh
+.buildkite/scripts/download_build_artifacts.sh
+
+export JOB=kibana-fleet-cypress
+
+echo "--- Fleet Cypress tests"
+
+cd "$XPACK_DIR"
+
+checks-reporter-with-killswitch "Fleet Cypress Tests" \
+ node scripts/functional_tests \
+   --debug --bail \
+   --kibana-install-dir "$KIBANA_BUILD_LOCATION" \
+   --config test/fleet_cypress/cli_config.ts
diff --git a/.gitignore b/.gitignore
@@ -86,3 +86,8 @@ report.asciidoc
 /bazel
 /bazel-*
 .bazelrc.user
+
+elastic-agent-*
+fleet-server-*
+elastic-agent.yml
+fleet-server.yml
diff --git a/src/dev/typescript/projects.ts b/src/dev/typescript/projects.ts
@@ -63,6 +63,9 @@ export const PROJECTS = [
     name: 'apm/ftr_e2e',
     disableTypeCheck: true,
   }),
+  createProject('x-pack/plugins/fleet/cypress/tsconfig.json', {
+    name: 'fleet/cypress',
+  }),
 
   createProject('x-pack/plugins/uptime/e2e/tsconfig.json', {
     name: 'uptime/synthetics-e2e-tests',

diff --git a/test/scripts/jenkins_fleet_cypress.sh b/test/scripts/jenkins_fleet_cypress.sh
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+source test/scripts/jenkins_test_setup_xpack.sh
+
+echo " -> Running fleet cypress tests"
+cd "$XPACK_DIR"
+
+checks-reporter-with-killswitch "Fleet Cypress Tests" \
+ node scripts/functional_tests \
+   --debug --bail \
+   --kibana-install-dir "$KIBANA_INSTALL_DIR" \
+   --config test/fleet_cypress/cli_config.ts
+
+echo ""
+echo ""
diff --git a/vars/tasks.groovy b/vars/tasks.groovy
@@ -164,6 +164,14 @@ def functionalXpack(Map params = [:]) {
         task(kibanaPipeline.functionalTestProcess('xpack-UptimePlaywright', './test/scripts/jenkins_uptime_playwright.sh'))
       }
     }
+
+    whenChanged([
+      'x-pack/plugins/fleet/',
+    ]) {
+      if (githubPr.isPr()) {
+        task(kibanaPipeline.functionalTestProcess('xpack-FleetCypress', './test/scripts/jenkins_fleet_cypress.sh'))
+      }
+    }
 
   }
 }

diff --git a/x-pack/plugins/fleet/.gitignore b/x-pack/plugins/fleet/.gitignore
diff --git a/x-pack/plugins/fleet/cypress/README.md b/x-pack/plugins/fleet/cypress/README.md
@@ -0,0 +1,148 @@
+# Cypress Tests
+
+The `fleet/cypress` directory contains functional UI tests that execute using [Cypress](https://www.cypress.io/).
+
+## Running the tests
+
+There are currently three ways to run the tests, comprised of two execution modes and two target environments, which will be detailed below.
+
+### Execution modes
+
+#### Interactive mode
+
+When you run Cypress in interactive mode, an interactive runner is displayed that allows you to see commands as they execute while also viewing the application under test. For more information, please see [cypress documentation](https://docs.cypress.io/guides/core-concepts/test-runner.html#Overview).
+
+#### Headless mode
+
+A headless browser is a browser simulation program that does not have a user interface. These programs operate like any other browser, but do not display any UI. This is why meanwhile you are executing the tests on this mode you are not going to see the application under test. Just the output of the test is displayed on the terminal once the execution is finished.
+
+### Target environments
+
+#### FTR (CI)
+
+This is the configuration used by CI. It uses the FTR to spawn both a Kibana instance (http://localhost:5620) and an Elasticsearch instance (http://localhost:9220) with a preloaded minimum set of data (see preceding "Test data" section), and then executes cypress against this stack. You can find this configuration in `x-pack/test/fleet_cypress`
+
+### Test Execution: Examples
+
+#### FTR + Headless (Chrome)
+
+Since this is how tests are run on CI, this will likely be the configuration you want to reproduce failures locally, etc.
+
+```shell
+# bootstrap kibana from the project root
+yarn kbn bootstrap
+
+# build the plugins/assets that cypress will execute against
+node scripts/build_kibana_platform_plugins
+
+# launch the cypress test runner
+cd x-pack/plugins/fleet
+yarn cypress:run-as-ci
+```
+#### FTR + Interactive
+
+This is the preferred mode for developing new tests.
+
+```shell
+# bootstrap kibana from the project root
+yarn kbn bootstrap
+
+# build the plugins/assets that cypress will execute against
+node scripts/build_kibana_platform_plugins
+
+# launch the cypress test runner
+cd x-pack/plugins/fleet
+yarn cypress:open-as-ci
+```
+
+Alternatively, kibana test server can be started separately, to pick up changes in UI (e.g. change in data-test-subj selector)
+
+```
+# launch kibana test server
+node scripts/functional_tests_server --config x-pack/test/fleet_cypress/config.ts
+
+# launch cypress runner
+node scripts/functional_test_runner --config x-pack/test/fleet_cypress/visual_config.ts
+```
+
+Note that you can select the browser you want to use on the top right side of the interactive runner.
+
+## Folder Structure
+
+### integration/
+
+Cypress convention. Contains the specs that are going to be executed.
+
+### fixtures/
+
+Cypress convention. Fixtures are used as external pieces of static data when we stub responses.
+
+### plugins/
+
+Cypress convention. As a convenience, by default Cypress will automatically include the plugins file cypress/plugins/index.js before every single spec file it runs.
+
+### screens/
+
+Contains the elements we want to interact with in our tests.
+
+Each file inside the screens folder represents a screen in our application.
+
+### tasks/
+
+_Tasks_ are functions that may be reused across tests.
+
+Each file inside the tasks folder represents a screen of our application. 
+
+## Test data
+
+The data the tests need:
+
+- Is generated on the fly using our application APIs (preferred way)
+- Is ingested on the ELS instance using the `es_archive` utility
+
+### How to generate a new archive
+
+**Note:** As mentioned above, archives are only meant to contain external data, e.g. beats data. Due to the tendency for archived domain objects (rules, signals) to quickly become out of date, it is strongly suggested that you generate this data within the test, through interaction with either the UI or the API.
+
+We use es_archiver to manage the data that our Cypress tests need.
+
+1. Set up a clean instance of kibana and elasticsearch (if this is not possible, try to clean/minimize the data that you are going to archive).
+2. With the kibana and elasticsearch instance up and running, create the data that you need for your test.
+3. When you are sure that you have all the data you need run the following command from: `x-pack/plugins/fleet`
+
+```sh
+node ../../../scripts/es_archiver save <nameOfTheFolderWhereDataIsSaved> <indexPatternsToBeSaved>  --dir ../../test/fleet_cypress/es_archives --config ../../../test/functional/config.js --es-url http://<elasticsearchUsername>:<elasticsearchPassword>@<elasticsearchHost>:<elasticsearchPort>
+```
+
+Example:
+
+```sh
+node ../../../scripts/es_archiver save custom_rules ".kibana",".siem-signal*"  --dir ../../test/fleet_cypress/es_archives --config ../../../test/functional/config.js --es-url http://elastic:changeme@localhost:9220
+```
+
+Note that the command will create the folder if it does not exist.
+
+## Development Best Practices
+
+### Clean up the state 
+
+Remember to clean up the state of the test after its execution, typically with the `cleanKibana` function. Be mindful of failure scenarios, as well: if your test fails, will it leave the environment in a recoverable state?
+
+### Minimize the use of es_archive
+
+When possible, create all the data that you need for executing the tests using the application APIS or the UI.
+
+### Speed up test execution time
+
+Loading the web page takes a big amount of time, in order to minimize that impact, the following points should be
+taken into consideration until another solution is implemented:
+
+- Group the tests that are similar in different contexts.
+- For every context login only once, clean the state between tests if needed without re-loading the page.
+- All tests in a spec file must be order-independent.
+
+Remember that minimizing the number of times the web page is loaded, we minimize as well the execution time.
+
+## Linting
+
+Optional linting rules for Cypress and linting setup can be found [here](https://github.com/cypress-io/eslint-plugin-cypress#usage)
diff --git a/x-pack/plugins/fleet/cypress/cypress.json b/x-pack/plugins/fleet/cypress/cypress.json
@@ -0,0 +1,19 @@
+{
+  "baseUrl": "http://localhost:5620",
+  "defaultCommandTimeout": 60000,
+  "requestTimeout": 60000,
+  "responseTimetout": 60000,
+  "execTimeout": 120000,
+  "pageLoadTimeout": 120000,
+  "nodeVersion": "system",
+  "retries": {
+    "runMode": 2
+  },
+  "screenshotsFolder": "../../../target/kibana-fleet/cypress/screenshots",
+  "trashAssetsBeforeRuns": false,
+  "video": false,
+  "videosFolder": "../../../target/kibana-fleet/cypress/videos",
+  "viewportHeight": 900,
+  "viewportWidth": 1440,
+  "screenshotOnRunFailure": true
+}