Add pipeline for flaky test runner job #46740
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
💚 Build Succeeded |
💚 Build Succeeded |
This comment has been minimized.
This comment has been minimized.
brianseeders
force-pushed
the
flaky-test-runner
branch
from
5b4e38e to
b86426a
Compare
brianseeders
added
v6.8.4
v7.3.2
v7.4.0
v7.5.0
Team:Operations
|
Pinging @elastic/kibana-operations |
💚 Build Succeeded |
brianseeders
force-pushed
the
flaky-test-runner
branch
from
b86426a to
c0e9c28
Compare
💔 Build Failed |
brianseeders
force-pushed
the
flaky-test-runner
branch
from
c0e9c28 to
903c898
Compare
💔 Build Failed |
brianseeders
commented
Sep 30, 2019
|
retest |
💚 Build Succeeded |
brianseeders
force-pushed
the
flaky-test-runner
branch
from
903c898 to
65d26a8
Compare
brianseeders
commented
Oct 1, 2019
|
|
||
| def buildStatus = buildUtils.getBuildStatus() | ||
| if (buildStatus != 'SUCCESS' && buildStatus != 'ABORTED') { | ||
| node('flyweight') { |
There was a problem hiding this comment.
I did change this function a little bit so that a node isn't allocated unless an e-mail actually needs to go out... Otherwise, the code in this file is just moved over from the original Jenkinsfile
There was a problem hiding this comment.
Is this something we could just run on master?
There was a problem hiding this comment.
Is there actually a label set up for master? I didn't realize
💚 Build Succeeded |
💚 Build Succeeded |
brianseeders
added a commit
to brianseeders/kibana
that referenced
this pull request
Oct 4, 2019
* WIP Jenkinsfile for flaky test runner * Fix syntax * A few more jenkinsfile fixes * A few more fixes * Can't round numbers in Jenkins groovy sandbox apparently * More fixes * Only do build_kbn_tp_sample_panel_action once during flaky testing * Fix path and try setting a different JOB value * Keep track of flaky test failures * Introduce flaky test for testing * Some flaky test pipeline cleanup * Fix a couple of issues with flaky test failure tracking * Update flaky test runner build name/desc with metadata * Revert "Introduce flaky test for testing" This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32. * Try adding a local shared library * Move local library loading logic to kibana pipeline library * Move shared groovy code to shared library * Add missed file * Add ability to specify multiple agents for flaky test runner * Update kibana-pipeline-library version * Fix bug causing early exit for oss ciGroup1
brianseeders
added a commit
to brianseeders/kibana
that referenced
this pull request
Oct 4, 2019
* WIP Jenkinsfile for flaky test runner * Fix syntax * A few more jenkinsfile fixes * A few more fixes * Can't round numbers in Jenkins groovy sandbox apparently * More fixes * Only do build_kbn_tp_sample_panel_action once during flaky testing * Fix path and try setting a different JOB value * Keep track of flaky test failures * Introduce flaky test for testing * Some flaky test pipeline cleanup * Fix a couple of issues with flaky test failure tracking * Update flaky test runner build name/desc with metadata * Revert "Introduce flaky test for testing" This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32. * Try adding a local shared library * Move local library loading logic to kibana pipeline library * Move shared groovy code to shared library * Add missed file * Add ability to specify multiple agents for flaky test runner * Update kibana-pipeline-library version * Fix bug causing early exit for oss ciGroup1
brianseeders
added a commit
to brianseeders/kibana
that referenced
this pull request
Oct 4, 2019
* WIP Jenkinsfile for flaky test runner * Fix syntax * A few more jenkinsfile fixes * A few more fixes * Can't round numbers in Jenkins groovy sandbox apparently * More fixes * Only do build_kbn_tp_sample_panel_action once during flaky testing * Fix path and try setting a different JOB value * Keep track of flaky test failures * Introduce flaky test for testing * Some flaky test pipeline cleanup * Fix a couple of issues with flaky test failure tracking * Update flaky test runner build name/desc with metadata * Revert "Introduce flaky test for testing" This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32. * Try adding a local shared library * Move local library loading logic to kibana pipeline library * Move shared groovy code to shared library * Add missed file * Add ability to specify multiple agents for flaky test runner * Update kibana-pipeline-library version * Fix bug causing early exit for oss ciGroup1
brianseeders
added a commit
to brianseeders/kibana
that referenced
this pull request
Oct 4, 2019
* WIP Jenkinsfile for flaky test runner * Fix syntax * A few more jenkinsfile fixes * A few more fixes * Can't round numbers in Jenkins groovy sandbox apparently * More fixes * Only do build_kbn_tp_sample_panel_action once during flaky testing * Fix path and try setting a different JOB value * Keep track of flaky test failures * Introduce flaky test for testing * Some flaky test pipeline cleanup * Fix a couple of issues with flaky test failure tracking * Update flaky test runner build name/desc with metadata * Revert "Introduce flaky test for testing" This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32. * Try adding a local shared library * Move local library loading logic to kibana pipeline library * Move shared groovy code to shared library * Add missed file * Add ability to specify multiple agents for flaky test runner * Update kibana-pipeline-library version * Fix bug causing early exit for oss ciGroup1
brianseeders
added a commit
that referenced
this pull request
Oct 4, 2019
This reverts commit e894255.
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
Oct 7, 2019
… into console-token-iterator * 'console-token-iterator' of github.com:jloleysens/kibana: (184 commits) [functional/services] update webdriver lib and types (elastic#47381) Standardizing IconField implementation across the app (elastic#47196) Move ui/value_suggestions ⇒ NP data plugin (elastic#45762) Remove ui/persisted_log - Part 2 (elastic#47236) Update gulp related packages (elastic#47421) Update dependency idx to ^2.5.6 (elastic#47399) try running fewer jobs in parallel on the same worker (elastic#47403) Update webpack related packages (elastic#47402) Update jsonwebtoken related packages (elastic#47400) Update gulp related packages (major) (elastic#46665) Update dependency prettier to ^1.18.2 (elastic#47340) Update dependency @types/puppeteer to ^1.20.1 (elastic#47339) Update dependency @elastic/elasticsearch to ^7.4.0 (elastic#47338) Update dependency tar-fs to ^1.16.3 (elastic#47341) [Code] Code Integrator Component (elastic#47180) [Canvas][i18n] Sidebar (elastic#46090) Generate uuid in task Manager as Kibana uuid may not yet have been initialised [Code] Embedded Code Snippet Component (elastic#47183) Revert "Add pipeline for flaky test runner job (elastic#46740)" SearchSource: fix docvalue_fields and fields intersection logic (elastic#46724) ...
brianseeders
added a commit
to brianseeders/kibana
that referenced
this pull request
Oct 22, 2019
This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened
brianseeders
added a commit
that referenced
this pull request
Oct 25, 2019
spalger
pushed a commit
to spalger/kibana
that referenced
this pull request
Oct 31, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
brianseeders
added a commit
to brianseeders/kibana
that referenced
this pull request
Oct 31, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
brianseeders
added a commit
to brianseeders/kibana
that referenced
this pull request
Oct 31, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
brianseeders
added a commit
to brianseeders/kibana
that referenced
this pull request
Oct 31, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
brianseeders
added a commit
to brianseeders/kibana
that referenced
this pull request
Oct 31, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
brianseeders
added a commit
that referenced
this pull request
Nov 15, 2019
brianseeders
added a commit
to brianseeders/kibana
that referenced
this pull request
Nov 26, 2019
* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable
brianseeders
added a commit
that referenced
this pull request
Dec 2, 2019
* Re-add pipeline for flaky test runner job (#48781) * Revert "Revert "Add pipeline for flaky test runner job (#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable * Retry git clone up to 8 times before failing a build (#50734) * Flaky test runner improvements/fixes (#49367) * Allocate multiple flaky test runner agents as executions grow * WIP support for deleting kibana install dir during CI * Add setup script for testing scripts * Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner * Change flaky test runner worker processes from 8 to 12 * Add labels to shell scripts in Jenkins (#49657) * Change pipeline timeout from 3 hours to 2 hours (#51098) * Remove flaky pipeline step not relevant for 6.8 * Re-add failed_tests cli * Fix PARALLEL_PIPELINE_WORKER_INDEX var
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #46163
How to start the job: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/build
The job will execute the chosen test suite
NUMBER_EXECUTIONStimes, using 14 parallel processes on one machine. The build only happens once, then the suite is executed many times.Duplicate test results will show up, so if the same suite errors out 5 times, all 5 instances will show up.
Example: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/4
Example of flaky test failing multiple times: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/4/testReport/
End of console:
I was able to move a lot of Jenkinsfile code to a shared location, but it unfortunately required creating a
varsdirectory in the root of the kibana repo. This is required by Jenkins shared libraries.