Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> policy name validation doesn't allow empty policy name

[kibanamachine]

A test failed on a tracked branch

Error: thrown: "Exceeded timeout of 5000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."
    at /var/lib/buildkite-agent/builds/kb-n2-4-spot-0891049732a0e254/elastic/kibana-on-merge/kibana/x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation/policy_name_validation.test.ts:39:3
    at _dispatchDescribe (/var/lib/buildkite-agent/builds/kb-n2-4-spot-0891049732a0e254/elastic/kibana-on-merge/kibana/node_modules/jest-circus/build/index.js:67:26)
    at describe (/var/lib/buildkite-agent/builds/kb-n2-4-spot-0891049732a0e254/elastic/kibana-on-merge/kibana/node_modules/jest-circus/build/index.js:30:5)
    at Object.<anonymous> (/var/lib/buildkite-agent/builds/kb-n2-4-spot-0891049732a0e254/elastic/kibana-on-merge/kibana/x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation/policy_name_validation.test.ts:14:1)
    at Runtime._execModule (/var/lib/buildkite-agent/builds/kb-n2-4-spot-0891049732a0e254/elastic/kibana-on-merge/kibana/node_modules/jest-runtime/build/index.js:1299:24)
    at Runtime._loadModule (/var/lib/buildkite-agent/builds/kb-n2-4-spot-0891049732a0e254/elastic/kibana-on-merge/kibana/node_modules/jest-runtime/build/index.js:898:12)
    at Runtime.requireModule (/var/lib/buildkite-agent/builds/kb-n2-4-spot-0891049732a0e254/elastic/kibana-on-merge/kibana/node_modules/jest-runtime/build/index.js:746:10)
    at jestAdapter (/var/lib/buildkite-agent/builds/kb-n2-4-spot-0891049732a0e254/elastic/kibana-on-merge/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:106:13)
    at runTestInternal (/var/lib/buildkite-agent/builds/kb-n2-4-spot-0891049732a0e254/elastic/kibana-on-merge/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/var/lib/buildkite-agent/builds/kb-n2-4-spot-0891049732a0e254/elastic/kibana-on-merge/kibana/node_modules/jest-runner/build/runTest.js:472:34)

First failure: CI Build - main

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[spalger]

This is the third suite in the x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation/ directory to become flaky this week. In order to prevent the rest of the tests here from blocking more PRs I'm skipping all the tests in this directory and linking them all here.

@Dosant all this flakiness seems to have started after #140804 so I'm going to assign you and hope you can take a look at these failures and the failures in the linked issues.

Other two suites that went flaky 3 days ago:

• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> downsample interval validation should validate an interval is greater or multiple than previous phase interval #141160
• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> timing validation should validate that min_age is equal or greater than previous phase min_age #115307

Other failures in this suite:

• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> policy name validation doesn't allow policy name with space #141646
• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> policy name validation doesn't allow policy name with comma #141648
• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> policy name validation doesn't allow policy name that is already used #141647
• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> policy name validation doesn't allow policy name starting with underscore #141649

Skipped in main/8.5

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServicesUx)

[Dosant]

@spalger, thanks, looking.
There is no flaky test runner for jest tests, right? Any tips how to approach this? I've tried to run locally in a loop, but getting a consist successful result :(

Oh, looks like there is quite a bit of the flakiness history in these tests... just saw a long discussion here #115307 with @sebelga 😢

cc @alisonelizabeth @yuliacech, just fyi about all the skipped validation tests

[Dosant]

It looks like all the failures are really Exceeded timeout of 5000 ms for a test

• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> policy name validation doesn't allow empty policy name #141645
• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> downsample interval validation should validate an interval is greater or multiple than previous phase interval #141160
• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> timing validation should validate that min_age is equal or greater than previous phase min_age #115307

Other errors in the suites are following the Exceeded timeout of 5000 ms for a test error from a previous test. I checked that their state depends on previous tests side effects and they also fail if the previous test fails:

• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> policy name validation doesn't allow policy name with space #141646
• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> policy name validation doesn't allow policy name with comma #141648
• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> policy name validation doesn't allow policy name that is already used #141647
• Failing test: Jest Tests.x-pack/plugins/index_lifecycle_management/__jest__/client_integration/edit_policy/form_validation - <EditPolicy /> policy name validation doesn't allow policy name starting with underscore #141649

So these would be "fixed" if we'd fix previous tests that failed with a timeout.

I saw @spalger and @sebelga already discussed if we could increase the timeout for those tests #115307 (comment) . As a better alternative @spalger suggested to change those to be run as real integration tests instead of unit tests. @spalger I don't think I had experience with those. What are the implications? Do you know if this a question of only moving the tests and changing the configs or will tests have to be refactored? Could you point me somewhere to learn more? maybe a good similar example?

Everyone, would it be OK if we increase the timeout for those tests to unskip all of those? Note that we've indeed recently added more code/tests to that page #140804, #138748 which I'd assume would increase the chances of hitting the timeout if we were already balancing on the edge.

We could also create an issue to move and refactor those as a proper integration_tests. At this point I am not sure who would tackle that (@elastic/kibana-app-services or @elastic/platform-deployment-management). My current understanding @elastic/platform-deployment-management still is and will be the owner of the code, and we (@elastic/kibana-app-services) help with TSDB effort

[spalger]

@Dosant we don't have any existing docs but I added an incomplete "What are the different types of tests, when should we write which type?" entry to our documentation matrix so we'll get to it eventually.

For the meantime, the gist is that Jest integration tests are just in a different directory and have two main differences:

1. They never use parallelism, this allows them to access file system resources, launch services, etc. without needing to worry about conflicts with other tests
2. They are allowed to take their sweet time, the default timeout is currently 10 minutes.

Unit tests are really intended for testing one small piece of the code in isolation. Shared state between test/dependencies on previous tests, taking more than a few milliseconds to run, or integrating with many systems and running an entire simulated front-end application in Node are all examples of things that go beyond the scope of what a unit test is intended for.

In order to move these tests to be "integration tests" they should be in a directory named "integration_tests". These tests will run when you use node scripts/jest_integration <some_project_dir>.