Reduce Cypress test flakiness

[qreshi]

Signed-off-by: Mohammad Qureshi [email protected]

Description

The first test in the first suite being run by Cypress seems to sometimes fail on timeouts, even when the Dashboards endpoint has been verified to be reachable before tests start (this seems to be more frequent in the CIs, especially now that we're running so many). It's not ideal but this change adds retries to both specs to see if the retry can account for the flakiness to lead to more consistent runs.

Issues Resolved

[List any issues this PR will resolve]

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #574 (54e8c06) into main (8c73802) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main     #574   +/-   ##
=========================================
  Coverage     61.36%   61.36%           
  Complexity      112      112           
=========================================
  Files            73       73           
  Lines          2472     2472           
  Branches        265      265           
=========================================
  Hits           1517     1517           
  Misses          778      778           
  Partials        177      177           

Flag	Coverage Δ
opensearch-notifications	61.36% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[qreshi]

I'm sometimes seeing flaky CI failures on Mac unrelated to the written tests themselves.

One example is a timeout for the Cypress verification which seems to be related to this. The only proper solution here would be to upgrade to a newer version of Cypress where we can provide CYPRESS_VERIFY_TIMEOUT to override the default value. If I keep seeing Mac specific flaky issues, I'll probably remove it from the matrix for now until the CI failure noise can be reduced.

[qreshi]

The retries have fixed the "loading page" flakiness issue for the first test in the suite, however, I'm still seeing test flakiness in failing to find elements in the tests sometimes (seems to be more frequent in Windows, could be because it's just slower):

1) Test edit senders
       edits ses sender region:
     AssertionError: Timed out retrying after 60000ms: Expected to find element: `2`, but never found it. Queried from element: [ <input#_selection_column_GmCZ6YQBwst8bUDIbd1F-checkbox.euiCheckbox__input>, 1 more... ]
      at Context.eval (http://localhost:5601/__cypress/tests?p=.cypress\integration\email_senders_and_groups.spec.js:250:65)

  2) Test delete senders
       deletes ses senders:
     AssertionError: Timed out retrying after 60000ms: Expected to find element: `input[placeholder="delete"]`, but never found it.
      at ensureExistence (http://localhost:5601/__cypress/runner/cypress_runner.js:162649:25)
      at $Cy.ensureElExistence (http://localhost:5601/__cypress/runner/cypress_runner.js:162667:12)
      at ensureExistence (http://localhost:5601/__cypress/runner/cypress_runner.js:151782:23)
      at onFailFn (http://localhost:5601/__cypress/runner/cypress_runner.js:151820:9)
      at tryCatcher (http://localhost:5601/__cypress/runner/cypress_runner.js:10765:23)
      at Promise._settlePromiseFromHandler (http://localhost:5601/__cypress/runner/cypress_runner.js:8700:31)
      at Promise._settlePromise (http://localhost:5601/__cypress/runner/cypress_runner.js:8757:18)
      at Promise._settlePromise0 (http://localhost:5601/__cypress/runner/cypress_runner.js:8802:10)
      at Promise._settlePromises (http://localhost:5601/__cypress/runner/cypress_runner.js:8878:18)
      at _drainQueueStep (http://localhost:5601/__cypress/runner/cypress_runner.js:5472:12)
      at _drainQueue (http://localhost:5601/__cypress/runner/cypress_runner.js:5465:9)
      at ../../node_modules/bluebird/js/release/async.js.Async._drainQueues (http://localhost:5601/__cypress/runner/cypress_runner.js:5481:5)
      at Async.drainQueues (http://localhost:5601/__cypress/runner/cypress_runner.js:5351:14)

  3) Test create, edit and delete recipient group
       opens email addresses popup:
     AssertionError: Timed out retrying after 60000ms: Expected to find content: '1 more' within the element: [ <a.euiLink.euiLink--subdued.euiBreadcrumb.osdBreadcrumbs>, 1 more... ] but never did.
      at Context.eval (http://localhost:5601/__cypress/tests?p=.cypress\integration\email_senders_and_groups.spec.js:347:24)

  4) Test create, edit and delete recipient group
       deletes recipient groups:
     AssertionError: Timed out retrying after 60000ms: Expected to find content: 'No recipient groups to display' but never did.
      at Context.eval (http://localhost:5601/__cypress/tests?p=.cypress\integration\email_senders_and_groups.spec.js:361:51)

We'll need to revisit this and see if the Cypress test can be updated to fetch these elements in more dependable ways.

[qreshi]

For the time being, I'm removing MacOS from the dashboards test matrix and we'll revisit any existing Cypress test flakiness but will not block on getting the version bumps in after this change is backported.