Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removes archives #86537

Merged
merged 30 commits into from
Dec 28, 2020
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
3d7a451
removes signals extra sanity
MadameSheema Dec 16, 2020
b5a0e25
fixes signals cleaning
MadameSheema Dec 16, 2020
daf466d
cleans kibana before each test execution
MadameSheema Dec 16, 2020
aea2a41
upgrades cypress to version 6.1.0
MadameSheema Dec 16, 2020
2ef701a
enables cypress execution on jenkins
MadameSheema Dec 16, 2020
a9c0400
removes alerts archives
MadameSheema Dec 16, 2020
b7ad376
removes timeline alerts archive
MadameSheema Dec 16, 2020
3099793
custom edition wip
MadameSheema Dec 16, 2020
99c3688
Merge branch 'master' into removes-archives
MadameSheema Dec 17, 2020
2205516
finishes cleaning
MadameSheema Dec 18, 2020
828ab3f
fixes events viewer
MadameSheema Dec 18, 2020
611b5ba
adds missing file
MadameSheema Dec 18, 2020
af163ac
fixes tests
MadameSheema Dec 18, 2020
40e4504
fixes typecheck issue
MadameSheema Dec 19, 2020
e78c001
removes archives
MadameSheema Dec 19, 2020
7f8aa5f
Merge branch 'master' into removes-archives
MadameSheema Dec 19, 2020
0c4c485
updates readme
MadameSheema Dec 21, 2020
6704df6
Merge branch 'removes-archives' of github.com:MadameSheema/kibana int…
MadameSheema Dec 21, 2020
15a87ca
Merge branch 'master' into removes-archives
kibanamachine Dec 21, 2020
630409d
improves clean up
MadameSheema Dec 22, 2020
89fe2c0
final clean up
MadameSheema Dec 22, 2020
ef8393d
fixes type check issue
MadameSheema Dec 22, 2020
4d2390e
Merge branch 'master' into removes-archives
kibanamachine Dec 22, 2020
e42b392
skips failing tests
MadameSheema Dec 22, 2020
a8a4570
last touches
MadameSheema Dec 22, 2020
e91583b
Merge branch 'master' into removes-archives
kibanamachine Dec 22, 2020
1247931
skipping failing tests
MadameSheema Dec 22, 2020
fcd352a
Merge branch 'removes-archives' of github.com:MadameSheema/kibana int…
MadameSheema Dec 22, 2020
ee296ca
Merge branch 'master' into removes-archives
kibanamachine Dec 23, 2020
5e44d0d
Merge branch 'master' into removes-archives
kibanamachine Dec 27, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 4 additions & 7 deletions x-pack/plugins/security_solution/cypress/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,9 +179,9 @@ CYPRESS_BASE_URL=http(s)://<username>:<password>@<kbnUrl> CYPRESS_ELASTICSEARCH_

## Best Practices

### Clean up the state between tests
### Clean up the state

Remember to clean up the state of the test after its execution.
Remember to use the `cleanKibana` method before starting the execution of the test
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than leaving this as a task for the test writer, we could enforce it as a global beforeEach hook.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I'm not 100% sure. There are some cases where using before instead of beforeEach, save us execution time. So we need to take into consideration that if we move that to a global hook, we are going to make our tests more slow. Another reason is because I don't like how Cypress handles right now global hooks.

Anyway, this is a think we can investigate/work in a different PR :)


### Minimize the use of es_archive

Expand All @@ -192,15 +192,12 @@ When possible, create all the data that you need for executing the tests using t
Loading the web page takes a big amount of time, in order to minimize that impact, the following points should be
taken into consideration until another solution is implemented:

- Don't refresh the page for every test to clean the state of it.
- Instead, group the tests that are similar in different contexts.
- Group the tests that are similar in different contexts.
- For every context login only once, clean the state between tests if needed without re-loading the page.
- All tests in a spec file must be order-independent.
- If you need to reload the page to make the tests order-independent, consider to create a new context.


Remember that minimizing the number of times the web page is loaded, we minimize as well the execution time.


## Reporting

When Cypress tests are run on the command line via non visual mode
Expand Down
139 changes: 67 additions & 72 deletions x-pack/plugins/security_solution/cypress/integration/alerts.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
import { newRule } from '../objects/rule';
import {
ALERTS,
ALERTS_COUNT,
Expand All @@ -24,37 +25,35 @@ import {
waitForAlertsToBeLoaded,
markInProgressFirstAlert,
goToInProgressAlerts,
waitForAlertsIndexToBeCreated,
} from '../tasks/alerts';
import { removeSignalsIndex } from '../tasks/api_calls/rules';
import { createCustomRuleActivated } from '../tasks/api_calls/rules';
import { cleanKibana } from '../tasks/common';
import { esArchiverLoad, esArchiverUnload } from '../tasks/es_archiver';
import { waitForAlertsToPopulate } from '../tasks/create_new_rule';
import { loginAndWaitForPage } from '../tasks/login';
import { refreshPage } from '../tasks/security_header';

import { DETECTIONS_URL } from '../urls/navigation';

describe('Alerts', () => {
context('Closing alerts', () => {
beforeEach(() => {
cleanKibana();
removeSignalsIndex();
esArchiverLoad('alerts');
loginAndWaitForPage(DETECTIONS_URL);
});

afterEach(() => {
esArchiverUnload('alerts');
waitForAlertsPanelToBeLoaded();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little bit wary of moving these steps into a beforeEach because of the fail-fast behavior and potential for false negatives, but I think that the speed and DRYness of this approach make it an acceptable tradeoff.

waitForAlertsIndexToBeCreated();
createCustomRuleActivated(newRule);
refreshPage();
waitForAlertsToPopulate();
});

it('Closes and opens alerts', () => {
waitForAlertsPanelToBeLoaded();
waitForAlertsToBeLoaded();

const numberOfAlertsToBeClosed = 3;
cy.get(ALERTS_COUNT)
.invoke('text')
.then((numberOfAlerts) => {
cy.get(SHOWING_ALERTS).should('have.text', `Showing ${numberOfAlerts} alerts`);

const numberOfAlertsToBeClosed = 3;
selectNumberOfAlerts(numberOfAlertsToBeClosed);

cy.get(SELECTED_ALERTS).should(
Expand All @@ -64,8 +63,6 @@ describe('Alerts', () => {

closeAlerts();
waitForAlerts();
cy.reload();
waitForAlerts();

const expectedNumberOfAlertsAfterClosing = +numberOfAlerts - numberOfAlertsToBeClosed;
cy.get(ALERTS_COUNT).should('have.text', expectedNumberOfAlertsAfterClosing.toString());
Expand All @@ -92,11 +89,6 @@ describe('Alerts', () => {

openAlerts();
waitForAlerts();
cy.reload();
waitForAlertsToBeLoaded();
waitForAlerts();
goToClosedAlerts();
waitForAlerts();

const expectedNumberOfClosedAlertsAfterOpened = 2;
cy.get(ALERTS_COUNT).should(
Expand Down Expand Up @@ -124,8 +116,6 @@ describe('Alerts', () => {
});

it('Closes one alert when more than one opened alerts are selected', () => {
waitForAlertsToBeLoaded();

cy.get(ALERTS_COUNT)
.invoke('text')
.then((numberOfAlerts) => {
Expand All @@ -137,8 +127,6 @@ describe('Alerts', () => {
cy.get(TAKE_ACTION_POPOVER_BTN).should('not.have.attr', 'disabled');

closeFirstAlert();
cy.reload();
waitForAlertsToBeLoaded();
waitForAlerts();

const expectedNumberOfAlerts = +numberOfAlerts - numberOfAlertsToBeClosed;
Expand All @@ -164,73 +152,82 @@ describe('Alerts', () => {
context('Opening alerts', () => {
beforeEach(() => {
cleanKibana();
removeSignalsIndex();
esArchiverLoad('closed_alerts');
loginAndWaitForPage(DETECTIONS_URL);
});
waitForAlertsPanelToBeLoaded();
waitForAlertsIndexToBeCreated();
createCustomRuleActivated(newRule);
refreshPage();
waitForAlertsToPopulate();
selectNumberOfAlerts(5);

cy.get(SELECTED_ALERTS).should('have.text', `Selected 5 alerts`);

afterEach(() => {
esArchiverUnload('closed_alerts');
closeAlerts();
waitForAlerts();
refreshPage();
});

it('Open one alert when more than one closed alerts are selected', () => {
waitForAlerts();
goToClosedAlerts();
waitForAlertsToBeLoaded();
waitForAlertsToPopulate();

cy.get(ALERTS_COUNT)
.invoke('text')
.then((numberOfAlerts) => {
const numberOfAlertsToBeOpened = 1;
const numberOfAlertsToBeSelected = 3;

cy.get(TAKE_ACTION_POPOVER_BTN).should('have.attr', 'disabled');
selectNumberOfAlerts(numberOfAlertsToBeSelected);
cy.get(TAKE_ACTION_POPOVER_BTN).should('not.have.attr', 'disabled');

openFirstAlert();
cy.reload();
.then((numberOfOpenedAlertsText) => {
const numberOfOpenedAlerts = parseInt(numberOfOpenedAlertsText, 10);
goToClosedAlerts();
waitForAlertsToBeLoaded();
waitForAlerts();

const expectedNumberOfAlerts = +numberOfAlerts - numberOfAlertsToBeOpened;
cy.get(ALERTS_COUNT).should('have.text', expectedNumberOfAlerts.toString());
cy.get(SHOWING_ALERTS).should(
'have.text',
`Showing ${expectedNumberOfAlerts.toString()} alerts`
);

goToOpenedAlerts();
waitForAlerts();

cy.get(ALERTS_COUNT).should('have.text', numberOfAlertsToBeOpened.toString());
cy.get(SHOWING_ALERTS).should(
'have.text',
`Showing ${numberOfAlertsToBeOpened.toString()} alert`
);
cy.get(ALERTS).should('have.length', numberOfAlertsToBeOpened);
cy.get(ALERTS_COUNT)
.invoke('text')
.then((numberOfAlerts) => {
const numberOfAlertsToBeOpened = 1;
const numberOfAlertsToBeSelected = 3;

cy.get(TAKE_ACTION_POPOVER_BTN).should('have.attr', 'disabled');
selectNumberOfAlerts(numberOfAlertsToBeSelected);
cy.get(SELECTED_ALERTS).should(
'have.text',
`Selected ${numberOfAlertsToBeSelected} alerts`
);

cy.get(TAKE_ACTION_POPOVER_BTN).should('not.have.attr', 'disabled');

openFirstAlert();
waitForAlerts();

const expectedNumberOfAlerts = +numberOfAlerts - numberOfAlertsToBeOpened;
cy.get(ALERTS_COUNT).should('have.text', expectedNumberOfAlerts.toString());
cy.get(SHOWING_ALERTS).should(
'have.text',
`Showing ${expectedNumberOfAlerts.toString()} alerts`
);

goToOpenedAlerts();
waitForAlerts();

cy.get(ALERTS_COUNT).should(
'have.text',
(numberOfOpenedAlerts + numberOfAlertsToBeOpened).toString()
);
cy.get(SHOWING_ALERTS).should(
'have.text',
`Showing ${(numberOfOpenedAlerts + numberOfAlertsToBeOpened).toString()} alerts`
);
});
});
});
});

context('Marking alerts as in-progress', () => {
beforeEach(() => {
cleanKibana();
removeSignalsIndex();
esArchiverLoad('alerts');
loginAndWaitForPage(DETECTIONS_URL);
});

afterEach(() => {
esArchiverUnload('alerts');
removeSignalsIndex();
waitForAlertsPanelToBeLoaded();
waitForAlertsIndexToBeCreated();
createCustomRuleActivated(newRule);
refreshPage();
waitForAlertsToPopulate();
});

it('Mark one alert in progress when more than one open alerts are selected', () => {
waitForAlerts();
waitForAlertsToBeLoaded();

cy.get(ALERTS_COUNT)
.invoke('text')
.then((numberOfAlerts) => {
Expand All @@ -242,8 +239,6 @@ describe('Alerts', () => {
cy.get(TAKE_ACTION_POPOVER_BTN).should('not.have.attr', 'disabled');

markInProgressFirstAlert();
cy.reload();
goToOpenedAlerts();
waitForAlertsToBeLoaded();

const expectedNumberOfAlerts = +numberOfAlerts - numberOfAlertsToBeMarkedInProgress;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import {
} from '../tasks/login';
import { waitForAlertsIndexToBeCreated } from '../tasks/alerts';
import { goToRuleDetails } from '../tasks/alerts_detection_rules';
import { createCustomRule, deleteCustomRule, removeSignalsIndex } from '../tasks/api_calls/rules';
import { createCustomRule, deleteCustomRule } from '../tasks/api_calls/rules';
import { getCallOut, waitForCallOutToBeShown, dismissCallOut } from '../tasks/common/callouts';
import { cleanKibana } from '../tasks/common';

Expand All @@ -42,7 +42,6 @@ describe('Detections > Callouts indicating read-only access to resources', () =>
// First, we have to open the app on behalf of a priviledged user in order to initialize it.
// Otherwise the app will be disabled and show a "welcome"-like page.
cleanKibana();
removeSignalsIndex();
loginAndWaitForPageWithoutDateRange(DETECTIONS_URL, ROLES.platform_engineer);
waitForAlertsIndexToBeCreated();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import {
goToOpenedAlerts,
waitForAlertsIndexToBeCreated,
} from '../tasks/alerts';
import { createCustomRule, removeSignalsIndex } from '../tasks/api_calls/rules';
import { createCustomRule } from '../tasks/api_calls/rules';
import { goToRuleDetails } from '../tasks/alerts_detection_rules';
import { waitForAlertsToPopulate } from '../tasks/create_new_rule';
import { esArchiverLoad, esArchiverUnload } from '../tasks/es_archiver';
Expand All @@ -35,11 +35,10 @@ import { refreshPage } from '../tasks/security_header';
import { DETECTIONS_URL } from '../urls/navigation';
import { cleanKibana } from '../tasks/common';

describe.skip('Exceptions', () => {
describe('Exceptions', () => {
const NUMBER_OF_AUDITBEAT_EXCEPTIONS_ALERTS = '1';
beforeEach(() => {
cleanKibana();
removeSignalsIndex();
loginAndWaitForPageWithoutDateRange(DETECTIONS_URL);
waitForAlertsIndexToBeCreated();
createCustomRule(newRule);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,12 @@
* you may not use this file except in compliance with the Elastic License.
*/
import {
FIFTH_RULE,
FIRST_RULE,
RULE_NAME,
RULE_SWITCH,
SECOND_RULE,
SEVENTH_RULE,
RULE_AUTO_REFRESH_IDLE_MODAL,
FOURTH_RULE,
} from '../screens/alerts_detection_rules';

import {
Expand All @@ -28,43 +27,45 @@ import {
waitForLoadElasticPrebuiltDetectionRulesTableToBeLoaded,
waitForRuleToBeActivated,
} from '../tasks/alerts_detection_rules';
import { esArchiverLoad, esArchiverUnload } from '../tasks/es_archiver';
import { loginAndWaitForPageWithoutDateRange } from '../tasks/login';
import { DEFAULT_RULE_REFRESH_INTERVAL_VALUE } from '../../common/constants';

import { DETECTIONS_URL } from '../urls/navigation';
import { removeSignalsIndex } from '../tasks/api_calls/rules';
import { createCustomRule, removeSignalsIndex } from '../tasks/api_calls/rules';
import { cleanKibana } from '../tasks/common';
import { existingRule, newOverrideRule, newRule, newThresholdRule } from '../objects/rule';

describe('Alerts detection rules', () => {
before(() => {
beforeEach(() => {
cleanKibana();
removeSignalsIndex();
esArchiverLoad('prebuilt_rules_loaded');
loginAndWaitForPageWithoutDateRange(DETECTIONS_URL);
waitForAlertsPanelToBeLoaded();
waitForAlertsIndexToBeCreated();
createCustomRule(newRule, 'rule1');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯 this is so much better, it's immediately clear that we're working with four rules in this test.

createCustomRule(existingRule, 'rule2');
createCustomRule(newOverrideRule, 'rule3');
createCustomRule(newThresholdRule, 'rule4');
});

after(() => {
esArchiverUnload('prebuilt_rules_loaded');
cy.clock().invoke('restore');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this teardown is specific to a single time-traveling test. It would be nice if such tests were isolated in their own context with clock-specific setup/teardown hooks.

});

it('Sorts by activated rules', () => {
loginAndWaitForPageWithoutDateRange(DETECTIONS_URL);
waitForAlertsPanelToBeLoaded();
waitForAlertsIndexToBeCreated();
goToManageAlertsDetectionRules();
waitForLoadElasticPrebuiltDetectionRulesTableToBeLoaded();

cy.get(RULE_NAME)
.eq(FIFTH_RULE)
.eq(SECOND_RULE)
.invoke('text')
.then((fifthRuleName) => {
activateRule(FIFTH_RULE);
.then((secondInitialRuleName) => {
activateRule(SECOND_RULE);
waitForRuleToBeActivated();
cy.get(RULE_NAME)
.eq(SEVENTH_RULE)
.eq(FOURTH_RULE)
.invoke('text')
.then((seventhRuleName) => {
activateRule(SEVENTH_RULE);
.then((fourthInitialRuleName) => {
activateRule(FOURTH_RULE);
waitForRuleToBeActivated();
sortByActivatedRules();
cy.get(RULE_NAME)
Expand All @@ -76,8 +77,8 @@ describe('Alerts detection rules', () => {
.invoke('text')
.then((secondRuleName) => {
const expectedRulesNames = `${firstRuleName} ${secondRuleName}`;
cy.wrap(expectedRulesNames).should('include', fifthRuleName);
cy.wrap(expectedRulesNames).should('include', seventhRuleName);
cy.wrap(expectedRulesNames).should('include', secondInitialRuleName);
cy.wrap(expectedRulesNames).should('include', fourthInitialRuleName);
});
});
cy.get(RULE_SWITCH).eq(FIRST_RULE).should('have.attr', 'role', 'switch');
Expand Down
Loading