Re-enable Firefox functional testing on CI #64159
Conversation
|
@elasticmachine merge upstream |
dmlemeshko
added
release_note:skip
There was a problem hiding this comment.
@dmlemeshko could you explain how you decided which of the ML smoke tags to remove (your PR currently removes 10 of the 16 smoke tags). Also, we faced some Firefox flakiness with ML tests a while back and as a result, currently all ML tests (except one small navigation test) have the skipFirefox tag attached on a top-level suite, so removing the smoke tag shouldn't make a difference.
To be clear: I'm fine with removing the smoke tag from the ML tests as I would have to re-evaluate Firefox stability for our tests step by step anyway before I re-enable some of them for Firefox execution. I just like to know some more details.
Removing the smoke from the transform tests is fine for now.
@pheyos I should start saying that The initial agreement was to keep Firefox worker run up to 50 min. So ideally no more suites should be tagged with smoke, unless we add another worker for Firefox. Because of this OOM issue we are limited up 20-25 min of run before Firefox consumes 4+ GB memory and the process got killed. Finally answering the question: I was cutting smoke tags based on execution time decrease and trying to keep balance between ML and other tests since 20 min is quite small amount. If you think some tests are more important we can replace them, I'm also thinking about replacing
I think even after Mozilla fixes the issue, we won't be able to run all the suites labelled @LeeDr We might need to put details about tags in docs/firefox config to make things clear for everyone, what do you think? |
dmlemeshko
force-pushed
the
re-enable-firefox-tests
branch
from
884ebc6 to
dbdcb44
Compare
|
Thanks for the details here and in our chat on slack @dmlemeshko! |
| @@ -154,16 +154,16 @@ A test suite is a collection of tests defined by calling `describe()`, and then | |||
| Use tags in `describe()` function to group functional tests. Tags include: | |||
| * `ciGroup{id}` - Assigns test suite to a specific CI worker | |||
| * `skipCloud` and `skipFirefox` - Excludes test suite from running on Cloud or Firefox | |||
| * `smoke` - Groups tests that run on Chrome and Firefox | |||
| * `includeFirefox` - Groups tests that run on Chrome and Firefox | |||
There was a problem hiding this comment.
++ I like the change from smoke to includeFirefox
|
@elasticmachine merge upstream |
💚 Build SucceededHistory
To update your PR or re-run it, just comment with: |
There was a problem hiding this comment.
This seems okay, we'll just watch for failures and potentially disable again if it causes a lot of flakiness.
I can't think of any places where removing the smoke tags will cause a problem... I checked with Liza and she said she doesn't use them.
Pinging @spalger as well since he is the one that disabled the tests
* bring testing on Firefox back * disable some tests * skip more tests * cut more suites for Firefox * skip more tests for Firefox * replace smoke tag with includeFirefox Co-authored-by: Elastic Machine <[email protected]>
* bring testing on Firefox back * disable some tests * skip more tests * cut more suites for Firefox * skip more tests for Firefox * replace smoke tag with includeFirefox Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
Summary
Closes #64128
Firefox has an open OMM issue , not allowing us to run the original
smoketagged test suites.With this PR we cut number of suites till things are stable on CI, in oder to have at least some automated testing in Firefox.
Flakiness check:
oss:firefoxSmoke 25 passed
oss:firefoxSmoke 25 passed
oss:firefoxSmoke 20 passed
xpack:firefoxSmoke 25 passed
xpack:firefoxSmoke 25 passed
xpack:firefoxSmoke 20 passed
In separate PR I will give a chance with starting session for each test suite, check if it helps to run more tests on Firefox and how much time longer execution is taking.