[SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting

x-pack/legacy/plugins/siem/cypress.json

@@ -1,3 +1,7 @@
 {
-  "baseUrl": "http://localhost:5601"
+  "baseUrl": "http://localhost:5601",
+  "screenshotsFolder": "../../../../target/kibana-siem/cypress/screenshots",
+  "trashAssetsBeforeRuns": false,
+  "video": false,
+  "videosFolder": "../../../../target/kibana-siem/cypress/videos"
 }

x-pack/legacy/plugins/siem/cypress/README.md

@@ -1,70 +1,242 @@
 # Cypress Tests
 
-The `siem/cypress` directory contains end to end tests (specific to the `SIEM` app) that execute via [Cypress](https://www.cypress.io/).
+The `siem/cypress` directory contains end to end tests, (plus a few tests
+that rely on mocked API calls), that execute via [Cypress](https://www.cypress.io/).
 
-At present, these tests are only executed in a local development environment; they are **not** integrated in the Kibana CI infrastructure, and therefore do **not** run automatically when you submit a PR.
+Cypress tests may be run against:
 
-See the `Server and Authentication Requirements` section below for additional details.
+- A local Kibana instance, interactively or via the command line. Credentials
+are specified via `kibna.dev.yml` or environment variables.
+- A remote Elastic Cloud instance (override `baseUrl`), interactively or via
+the command line. Again, credentials are specified via `kibna.dev.yml` or
+environment variables.
+- As part of CI (override `baseUrl` and pass credentials via the
+`CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
+environment variables), via command line.
 
-## Organizing Tests and (Mock) Data
+At present, Cypress tests are only executed manually. They are **not** yet
+integrated in the Kibana CI infrastructure, and therefore do **not** run
+automatically when you submit a PR.
 
-- Code and CSS selectors that may be re-used across tests should be added to `siem/cypress/integration/lib`, as described below
-- Smoke Tests are located in `siem/cypress/integration/smoke_tests`
-- Mocked responses from the server are located in `siem/cypress/fixtures`
+## Smoke Tests
 
-### `cypress/integration/lib`
+Smoke Tests are located in `siem/cypress/integration/smoke_tests`
 
-The `cypress/integration/lib` folder contains code intended to be re-used across many different tests.
+## Test Helpers
 
-- Files named `helpers.ts` (e.g. `siem/cypress/integration/lib/login/helpers.ts`) contain functions (e.g. `login`) that may be imported and invoked from multiple tests.
+_Test helpers_ are functions that may be re-used across tests.
 
-- Files named `selectors.ts` export CSS selectors for re-use. For example, `siem/cypress/integration/lib/login/selectors.ts` exports the following selector that matches the Username text area in the Kibana login page:
+- Reusable code and CSS selectors should be added to
+`siem/cypress/integration/lib`, as described below.
 
-```
+### Reusable Test Helper Functions and CSS Selectors
+
+The `cypress/integration/lib` directory contains code intended to be re-used
+across many different tests. Add reusable test helper functions and CSS
+selectors to directories under `cypress/integration/lib`.
+
+- Files named `helpers.ts` (e.g. `siem/cypress/integration/lib/login/helpers.ts`)
+contain functions (e.g. `login`) that may be imported and invoked from multiple tests.
+
+- Files named `selectors.ts` export CSS selectors for re-use. For example,
+`siem/cypress/integration/lib/login/selectors.ts` exports the following selector
+that matches the Username text area in the Kibana login page:
+
+```sh
 export const USERNAME = '[data-test-subj="loginUsername"]';
 ```
 
-## Server and Authentication Requirements
+## Mock Data
+
+We prefer not to mock API responses in most of our smoke tests, but sometimes
+it's necessary because a test must assert that a specific value is rendered,
+and it's not possible to derive that value based on the data in the
+envrionment where tests are running.
+
+Mocked responses API from the server are located in `siem/cypress/fixtures`.
+
+## Authentication
+
+When running tests, there are two ways to specify the credentials used to
+authenticate with Kibana:
+
+- Via `kibana.dev.yml` (recommended for developers)
+- Via the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
+environment variables (recommended for CI), or when testing a remote Kibana
+instance, e.g. in Elastic Cloud.
+
+Note: Tests that use the `login()` test helper function for authentication will
+automatically use the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
+environment variables when they are defined, and fall back to the values in
+`config/kibana.dev.yml` when they are unset.
+
+### Content Security Policy (CSP) Settings
+
+Your local or cloud Kibana server must have the `csp.strict: false` setting
+configured in `kibana.dev.yml`, or `kibana.yml`, as shown in the example below:
+
+```yaml
+csp.strict: false
+```
+
+The above setting is required to prevent the _Please upgrade
+your browser_ / _This Kibana installation has strict security requirements
+enabled that your current browser does not meet._ warning that's displayed for
+unsupported user agents, like the one reported by Cypress when running tests.
+
+### Example `kibana.dev.yml`
 
-The current version of the Smoke Tests require running a local Kibana server that connects to an instance of `elasticsearch`. A file named `config/kibana.dev.yml` like the example below is required to run the tests:
+If you're a developer running tests interactively or on the command line, the
+easiset way to specify the credentials used for authentication is to update
+ `kibana.dev.yml` per the following example:
 
 ```yaml
+csp.strict: false
 elasticsearch:
   username: 'elastic'
   password: '<password>'
   hosts: ['https://<server>:9200']
 ```
 
-The `username` and `password` from `config/kibana.dev.yml` will be read by the `login` test helper function when tests authenticate with Kibana.
+## Running Tests Interactively
 
-See the `Running Tests Interactively` section for details.
+Use the Cypress interactive test runner to develop and debug specific tests
+by adding a `.only` to the test you're developing, or click on a specific
+spec in the interactive test runner to run just the tests in that spec.
 
-### Content Security Policy (CSP) Settings
+To run and debug tests in interactively via the Cypress test runner:
 
- Your local or cloud Kibana server must have the `csp.strict: false` setting
-configured in `kibana.dev.yml`, or `kibana.yml`, as shown in the example below:
+1. Disable CSP on the local or remote Kibana instance, as described in the
+_Content Security Policy (CSP) Settings_ section above.
 
- ```yaml
-csp.strict: false
+2. To specify the credentials required for authentication, configure
+`config/kibana.dev.yml`, as described in the _Server and Authentication
+Requirements_ section above, or specify them via environment variables
+as described later in this section.
+
+3. Start a local instance of the Kibana development server (only if testing against a
+local host):
+
+```sh
+yarn start --no-base-path
 ```
 
-## Running Tests Interactively
+4. Launch the Cypress interactive test runner via one of the following options:
 
-To run tests in interactively via the Cypress test runner:
+- To run tests interactively against the default (local) host specified by
+`baseUrl`, as configured in `plugins/siem/cypress.json`:
 
-1. Create and configure a `config/kibana.dev.yml`, as described in the `Server and Authentication Requirements` section above.
+```sh
+cd x-pack/legacy/plugins/siem
+yarn cypress:open
+```
+
+- To (optionally) run tests interactively against a different host, pass the
+`CYPRESS_baseUrl` environment variable on the command line when launching the
+test runner, as shown in the following example:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 yarn cypress:open
+```
 
-2. Start a local instance of the Kibana development server:
+- To (optionally) override username and password via environment variables when
+running tests interactively:
 
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open
 ```
+
+5. Click the `Run all specs` button in the Cypress test runner (after adding
+a `.only` to an `it` or `describe` block).
+
+## Running (Headless) Tests on the Command Line
+
+To run (headless) tests on the command line:
+
+1. Disable CSP on the local or remote Kibana instance, as described in the
+_Content Security Policy (CSP) Settings_ section above.
+
+2. To specify the credentials required for authentication, configure
+`config/kibana.dev.yml`, as described in the _Server and Authentication
+Requirements_ section above, or specify them via environment variables
+as described later in this section.
+
+3. Start a local instance of the Kibana development server (only if testing against a
+local host):
+
+```sh
 yarn start --no-base-path
 ```
 
-3. Launch the Cypress interactive test runner:
+4. Launch the Cypress command line test runner via one of the following options:
+
+- To run tests on the command line against the default (local) host specified by
+`baseUrl`, as configured in `plugins/siem/cypress.json`:
 
 ```sh
 cd x-pack/legacy/plugins/siem
-yarn cypress:open
+yarn cypress:run
+```
+
+- To (optionally) run tests on the command line against a different host, pass
+`CYPRESS_baseUrl` as an environment variable on the command line, as shown in
+the following example:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 yarn cypress:run
 ```
 
-4. Click the `Run all specs` button in the Cypress test runner
+- To (optionally) override username and password via environment variables when
+running via the command line:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run
+```
+
+## Reporting
+
+When Cypress tests are run on the command line via `yarn cypress:run`,
+reporting artifacts are generated under the `target` directory in the root
+of the Kibana, as detailed for each artifact type in the sections bleow.
+
+### HTML Reports
+
+An HTML report (e.g. for email notifications) is output to:
+
+```
+target/kibana-siem/cypress/results/output.html
+```
+
+### Screenshots
+
+Screenshots of failed tests are output to:
+
+```
+target/kibana-siem/cypress/screenshots
+```
+
+### `junit` Reports
+
+The Kibana CI process reports `junit` test results from the `target/junit` directory.
+
+Cypress `junit` reports are generated in `target/kibana-siem/cypress/results`
+and copied to the `target/junit` directory.
+
+### Videos (optional)
+
+Videos are disabled by default, but can optionally be enabled by setting the
+`CYPRESS_video=true` environment variable:
+
+```
+CYPRESS_video=true yarn cypress:run
+```
+
+Videos are (optionally) output to:
+
+```
+target/kibana-siem/cypress/videos
+```

x-pack/legacy/plugins/siem/cypress/integration/smoke_tests/fields_browser/fields_browser.spec.ts

@@ -21,7 +21,7 @@ import {
 } from '../../lib/fields_browser/selectors';
 import { logout } from '../../lib/logout';
 import { HOSTS_PAGE } from '../../lib/urls';
-import { loginAndWaitForPage } from '../../lib/util/helpers';
+import { loginAndWaitForPage, DEFAULT_TIMEOUT } from '../../lib/util/helpers';
 
 const defaultHeaders = [
   { id: '@timestamp' },
@@ -200,9 +200,9 @@ describe('Fields Browser', () => {
 
     cy.get(`[data-test-subj="headers-group"]`).then(headersDropArea => drop(headersDropArea));
 
-    clickOutsideFieldsBrowser();
-
-    cy.get(`[data-test-subj="header-text-${toggleField}"]`).should('exist');
+    cy.get(`[data-test-subj="header-text-${toggleField}"]`, { timeout: DEFAULT_TIMEOUT }).should(
+      'exist'
+    );
   });
 
   it('resets all fields in the timeline when `Reset Fields` is clicked', () => {

x-pack/legacy/plugins/siem/cypress/integration/smoke_tests/timeline/toggle_column.spec.ts

@@ -9,7 +9,7 @@ import { populateTimeline } from '../../lib/fields_browser/helpers';
 import { logout } from '../../lib/logout';
 import { toggleFirstEventDetails } from '../../lib/timeline/helpers';
 import { HOSTS_PAGE } from '../../lib/urls';
-import { loginAndWaitForPage } from '../../lib/util/helpers';
+import { loginAndWaitForPage, DEFAULT_TIMEOUT } from '../../lib/util/helpers';
 
 describe('toggle column in timeline', () => {
   beforeEach(() => {
@@ -72,6 +72,8 @@ describe('toggle column in timeline', () => {
 
     cy.get(`[data-test-subj="headers-group"]`).then(headersDropArea => drop(headersDropArea));
 
-    cy.get(`[data-test-subj="header-text-${idField}"]`).should('exist');
+    cy.get(`[data-test-subj="header-text-${idField}"]`, { timeout: DEFAULT_TIMEOUT }).should(
+      'exist'
+    );
   });
 });

x-pack/legacy/plugins/siem/package.json

@@ -6,20 +6,37 @@
   "license": "Elastic-License",
   "scripts": {
     "build-graphql-types": "node scripts/generate_types_from_graphql.js",
-    "cypress:open": "cypress open"
+    "cypress:open": "cypress open",
+    "cypress:run": "cypress run --spec ./cypress/integration/**/*.spec.ts --reporter mocha-multi-reporters --reporter-options configFile=./reporter_config.json; mochawesome-merge --reportDir ../../../../target/kibana-siem/cypress/results > ../../../../target/kibana-siem/cypress/results/output.json; marge ../../../../target/kibana-siem/cypress/results/output.json --reportDir ../../../../target/kibana-siem/cypress/results; mkdir -p ../../../../target/junit && cp ../../../../target/kibana-siem/cypress/results/*.xml ../../../../target/junit/"
   },
   "devDependencies": {
     "@cypress/webpack-preprocessor": "^4.1.0",
     "@types/js-yaml": "^3.12.1",
     "@types/lodash": "^4.14.110",
     "@types/react-beautiful-dnd": "^10.0.1",
-    "cypress": "^3.3.1",
+    "cypress": "^3.4.1",
     "js-yaml": "^3.13.1",
+    "mocha-junit-reporter": "^1.23.1",
+    "mocha-multi-reporters": "^1.1.7",
+    "mochawesome": "^4.0.1",
+    "mochawesome-merge": "^2.0.1",
+    "mochawesome-report-generator": "^4.0.1",
     "ts-loader": "^6.0.4"
   },
   "dependencies": {
     "lodash": "^4.17.13",
     "react-beautiful-dnd": "^10.0.1",
     "react-markdown": "^4.0.6"
+  },
+  "workspaces": {
+    "packages": [
+      "packages/*"
+    ],
+    "nohoist": [
+      "**/mochawesome",
+      "**/mochawesome/**",
+      "**/mocha-multi-reporters",
+      "**/mocha-multi-reporters/**"
+    ]
   }
-}
+}

x-pack/legacy/plugins/siem/reporter_config.json

@@ -0,0 +1,10 @@
+{
+  "reporterEnabled": "mochawesome, mocha-junit-reporter",
+  "reporterOptions": {
+    "html": false,
+    "json": true,
+    "mochaFile": "../../../../target/kibana-siem/cypress/results/results-[hash].xml",
+    "overwrite": false,
+    "reportDir": "../../../../target/kibana-siem/cypress/results"
+  }
+}