Skip to content

Commit

Permalink
[docs] Fix functional test doc formatting (elastic#169715)
Browse files Browse the repository at this point in the history
## Summary

Follow up to elastic#169596

---------

Co-authored-by: Christiane (Tina) Heiligers <[email protected]>
  • Loading branch information
adamkasztenny and TinaHeiligers authored Oct 24, 2023
1 parent 60a4f70 commit 6f30fa3
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion dev_docs/operations/writing_stable_functional_tests.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Consistently writing functional tests that aren't flaky is impossible. There are

When you watch tests execute locally it can be tempting to think "after I click this button I can click this other button" but this assumes that the first button click will always execute its click handler immediately, or that the render of the second button will be done immediately. The reality is that user interfaces are super complex and these types of assumptions are by far the most common cause of flakiness in tests.

We also have to remember that we can't assume a test passing locally will mean it will pass in CI. The two environments are different. There is a lot we could mention, but at a high level, most functional tests are run on 4 core 16 GB machines, and these machines are virtualized, which means neighbors can cause modest but variable levels of performance. Additionally, end-to-end tests in CI are run against {kib} distributions, using default memory configurations, while we run the tests under the `--dev` flag locally with, most likely, a different memory configuration.
We also have to remember that we can't assume a test passing locally will mean it will pass in CI. The two environments are different. There is a lot we could mention, but at a high level, most functional tests are run on 4 core 16 GB machines, and these machines are virtualized, which means neighbors can cause modest but variable levels of performance. Additionally, end-to-end tests in CI are run against Kibana distributions, using default memory configurations, while we run the tests under the `--dev` flag locally with, most likely, a different memory configuration.

There are all sorts of things that can happen to delay a click handler, or react render, and we need to be prepared for those in our tests. We can do this using appropriate timeouts for specific actions, retry logic, and validating our assumptions with code.

Expand Down

0 comments on commit 6f30fa3

Please sign in to comment.