[CI] allow for manual triggered cypress test jobs to accept inputs #5134

kavilla · 2023-09-27T12:03:33Z

Description

Utilizing the GitHub workflow_dispatch, we can trigger a job with a specific inputs.

By default, we run from this repo:
https://github.com/opensearch-project/opensearch-dashboards-functional-test

This workflow uses the target branch of the PR to pull down from the FTRepo and run the tests from that specific branch.

For example, PRs against main will pull down main from the FTRepo.

The problem occurs when new functionality is opened or a bug is fixed and, per industry standard, we want to see tests added to ensure functionality, stability, and raise the bar.

However, the cypress tests PR into the FTRepo depends on the new code to be merged for the CI within the FTRepo to work. So we have a stalemate that usually slows down PR review time.

Here, a manual run can be triggered by a maintainer. It will utilize the source branch from the PR if provided.

Issues Related

#4019

Screenshot

Unfortunately, GitHub only enables the getting of ref from the manual run. To do this, a maintainer will have to place inputs from:

Actions > Run cypress tests

Which you can enter inputs and trigger the job:

Results will show up like this if there is associated PR to the triggered run:

As you can see it ran from my own fork but i was able to run a different branch from my own fork of the functional test repo. This is like a manually verification still, I have not looked into updating the status of a PR but I believe we can set to pass still.

Testing the changes

I was playing around with this PR by changing my default.

Check List

codecov · 2023-09-27T12:25:10Z

Codecov Report

Merging #5134 (439537c) into main (46d7c2d) will increase coverage by 0.00%.
Report is 1 commits behind head on main.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #5134   +/-   ##
=======================================
  Coverage   66.50%   66.50%           
=======================================
  Files        3403     3403           
  Lines       65026    65026           
  Branches    10401    10401           
=======================================
+ Hits        43243    43245    +2     
+ Misses      19214    19213    -1     
+ Partials     2569     2568    -1

Flag	Coverage Δ
Linux_1	`34.82% <ø> (ø)`
Linux_2	`55.31% <ø> (ø)`
Linux_3	`44.61% <ø> (+<0.01%)`	⬆️
Linux_4	`34.89% <ø> (ø)`
Windows_1	`34.83% <ø> (ø)`
Windows_2	`55.27% <ø> (ø)`
Windows_3	`44.61% <ø> (ø)`
Windows_4	`34.89% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

Utilizing the GitHub `workflow_dispatch`, we can trigger a job with a specific inputs. By default, we run from this repo: https://github.com/opensearch-project/opensearch-dashboards-functional-test This workflow uses the target branch of the PR to pull down from the FTRepo and run the tests from that specific branch. For example, PRs against `main` will pull down `main` from the FTRepo. The problem occurs when new functionality is opened or a bug is fixed and, per industry standard, we want to see tests added to ensure functionality, stability, and raise the bar. However, the cypress tests PR into the FTRepo depends on the new code to be merged for the CI within the FTRepo to work. So we have a stalemate that usually slows down PR review time. Here, a manual run can be triggered by a maintainer. It will utilize the source branch from the PR if provided. Related to: opensearch-project#4019 Signed-off-by: Kawika Avilla <[email protected]>

AMoo-Miki · 2023-09-27T21:55:05Z

.github/workflows/cypress_workflow.yml

+  SOURCE_REPO: ${{ github.repository }}
+  SOURCE_BRANCH: ${{ github.base_ref }}
+  TEST_REPO: ${{ inputs.test_repo != '' && inputs.test_repo || 'opensearch-project/opensearch-dashboards-functional-test' }}
+  TEST_BRANCH: "${{ inputs.test_branch != '' && inputs.test_branch || github.base_ref }}"


If a PR is raised against a feature branch, github.base_ref would point to that branch; what if that branch doesn't exist on the FT repo?

PS, i see that we currently don't handle that situation anyway!

yeah. that's happened a couple of times in the pass.

i end up just creating the branch in the FTRepo. Part of the release improvements and artifacts we want to have fallback solutions.

joshuarrrr

cool

joshuarrrr · 2023-09-27T22:15:50Z

@kavilla Not to scope creep this, but some documentation about when and how to run this would be useful to be able to reference without coming back to this PR. I know we've talked about a PR review markdown doc, but it could go in one of our existing docs, too.

kavilla · 2023-09-27T22:30:20Z

@kavilla Not to scope creep this, but some documentation about when and how to run this would be useful to be able to reference without coming back to this PR. I know we've talked about a PR review markdown doc, but it could go in one of our existing docs, too.

Yeah I was thinking too because the inputs vs defaults I can see being misunderstood. I was originally thinking adding a statement to the CONTRIBUTING doc and will create a subsequent issue for a PR.md

kavilla · 2023-09-27T23:46:27Z

Just realized the cypress tests are failing for most PRs now.

Will look I know we addressing some with the Discover work. This will assist with seeing if PRs in the FTRepo so I will merge as is.

I have created a follow-up issue for docs: #5139

Utilizing the GitHub `workflow_dispatch`, we can trigger a job with a specific inputs. By default, we run from this repo: https://github.com/opensearch-project/opensearch-dashboards-functional-test This workflow uses the target branch of the PR to pull down from the FTRepo and run the tests from that specific branch. For example, PRs against `main` will pull down `main` from the FTRepo. The problem occurs when new functionality is opened or a bug is fixed and, per industry standard, we want to see tests added to ensure functionality, stability, and raise the bar. However, the cypress tests PR into the FTRepo depends on the new code to be merged for the CI within the FTRepo to work. So we have a stalemate that usually slows down PR review time. Here, a manual run can be triggered by a maintainer. It will utilize the source branch from the PR if provided. Related to: #4019 Signed-off-by: Kawika Avilla <[email protected]> (cherry picked from commit 34a8594) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md

github-actions · 2023-09-28T01:16:57Z

[MANUAL CYPRESS TEST RUN RESULTS]

❌ Cypress test run failed!

Inputs:

Source repo: opensearch-project/OpenSearch-Dashboards
Source branch: Test repo: `abbyhu2000/opensearch-dashboards-functional-test` Test branch:discover_app``

Link to results:

https://github.com/opensearch-project/OpenSearch-Dashboards/actions/runs/6332816838

Originally: opensearch-project#5134 I believed it to be a good idea to ensure that triggering the CI should always run a batch of tests. But I realized that some tests can be added and just be bad tests until updated. Or if we want to verify a flaky test in the CI. So instead of only making it additional tests, the spec input will complete replace the default tests (unless you append to the default). Empty will run the default spec still. This will append on to the PR comment with the spec that was run. So at least maintainers can see it passed for a set of tests but it did not run the ones we usually run. Also adding some formatting for empty values for the PR comment. Issue related: opensearch-project#4019 Signed-off-by: Kawika Avilla <[email protected]>

Originally: #5134 I believed it to be a good idea to ensure that triggering the CI should always run a batch of tests. But I realized that some tests can be added and just be bad tests until updated. Or if we want to verify a flaky test in the CI. So instead of only making it additional tests, the spec input will complete replace the default tests (unless you append to the default). Empty will run the default spec still. This will append on to the PR comment with the spec that was run. So at least maintainers can see it passed for a set of tests but it did not run the ones we usually run. Also adding some formatting for empty values for the PR comment. Issue related: #4019 Signed-off-by: Kawika Avilla <[email protected]>

Originally: #5134 I believed it to be a good idea to ensure that triggering the CI should always run a batch of tests. But I realized that some tests can be added and just be bad tests until updated. Or if we want to verify a flaky test in the CI. So instead of only making it additional tests, the spec input will complete replace the default tests (unless you append to the default). Empty will run the default spec still. This will append on to the PR comment with the spec that was run. So at least maintainers can see it passed for a set of tests but it did not run the ones we usually run. Also adding some formatting for empty values for the PR comment. Issue related: #4019 Signed-off-by: Kawika Avilla <[email protected]> (cherry picked from commit 6154e2d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Originally: #5134 I believed it to be a good idea to ensure that triggering the CI should always run a batch of tests. But I realized that some tests can be added and just be bad tests until updated. Or if we want to verify a flaky test in the CI. So instead of only making it additional tests, the spec input will complete replace the default tests (unless you append to the default). Empty will run the default spec still. This will append on to the PR comment with the spec that was run. So at least maintainers can see it passed for a set of tests but it did not run the ones we usually run. Also adding some formatting for empty values for the PR comment. Issue related: #4019 (cherry picked from commit 6154e2d) Signed-off-by: Kawika Avilla <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-project#5134 Incorrectly uses the source of the code to pull down to BASE REF, which would likely be `main` or `2.x`, etc. It should be pulling down the PR branch. Cypress tests at the time of merging were failing due to unrelated issue of disk allocation. Signed-off-by: Kawika Avilla <[email protected]>

opensearch-project#5134 Incorrectly uses the source of the code to pull down to BASE REF, which would likely be `main` or `2.x`, etc. It should be pulling down the PR branch. Cypress tests at the time of merging were failing due to unrelated issue of disk allocation. Not setting the env variables causes the workflow to rely on the default values which is a return back to the original implementation and if the env is set then it will be not empty. Signed-off-by: Kawika Avilla <[email protected]> do not set source env variables Signed-off-by: Kawika Avilla <[email protected]>

#5134 Incorrectly uses the source of the code to pull down to BASE REF, which would likely be `main` or `2.x`, etc. It should be pulling down the PR branch. Cypress tests at the time of merging were failing due to unrelated issue of disk allocation. Not setting the env variables causes the workflow to rely on the default values which is a return back to the original implementation and if the env is set then it will be not empty. Signed-off-by: Kawika Avilla <[email protected]>

#5134 Incorrectly uses the source of the code to pull down to BASE REF, which would likely be `main` or `2.x`, etc. It should be pulling down the PR branch. Cypress tests at the time of merging were failing due to unrelated issue of disk allocation. Not setting the env variables causes the workflow to rely on the default values which is a return back to the original implementation and if the env is set then it will be not empty. Signed-off-by: Kawika Avilla <[email protected]> (cherry picked from commit dc6a7ec) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

#5134 Incorrectly uses the source of the code to pull down to BASE REF, which would likely be `main` or `2.x`, etc. It should be pulling down the PR branch. Cypress tests at the time of merging were failing due to unrelated issue of disk allocation. Not setting the env variables causes the workflow to rely on the default values which is a return back to the original implementation and if the env is set then it will be not empty. (cherry picked from commit dc6a7ec) Signed-off-by: Kawika Avilla <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-project#5134 Incorrectly uses the source of the code to pull down to BASE REF, which would likely be `main` or `2.x`, etc. It should be pulling down the PR branch. Cypress tests at the time of merging were failing due to unrelated issue of disk allocation. Not setting the env variables causes the workflow to rely on the default values which is a return back to the original implementation and if the env is set then it will be not empty. Signed-off-by: Kawika Avilla <[email protected]> Signed-off-by: Willie Hung <[email protected]>

opensearch-project#5134 Incorrectly uses the source of the code to pull down to BASE REF, which would likely be `main` or `2.x`, etc. It should be pulling down the PR branch. Cypress tests at the time of merging were failing due to unrelated issue of disk allocation. Not setting the env variables causes the workflow to rely on the default values which is a return back to the original implementation and if the env is set then it will be not empty. Signed-off-by: Kawika Avilla <[email protected]>

github-actions bot added the distinguished-contributor label Sep 27, 2023

kavilla added ci and removed distinguished-contributor labels Sep 27, 2023

kavilla changed the title ~~[CI] allow for manual triggered jobs to accept inputs~~ [CI] allow for manual triggered cypress test jobs to accept inputs Sep 27, 2023

kavilla force-pushed the kavilla/ci/cypress_input branch from be8b517 to 17334b4 Compare September 27, 2023 21:18

kavilla marked this pull request as ready for review September 27, 2023 21:18

kavilla requested review from ananzh, seanneumann, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, kristenTian, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, BSFishy and curq as code owners September 27, 2023 21:18

kavilla added the backport 2.x label Sep 27, 2023

kavilla force-pushed the kavilla/ci/cypress_input branch from 17334b4 to 439537c Compare September 27, 2023 21:24

AMoo-Miki reviewed Sep 27, 2023

View reviewed changes

AMoo-Miki approved these changes Sep 27, 2023

View reviewed changes

joshuarrrr approved these changes Sep 27, 2023

View reviewed changes

kavilla merged commit 34a8594 into opensearch-project:main Sep 27, 2023
53 of 55 checks passed

opensearch-trigger-bot bot mentioned this pull request Sep 27, 2023

[Backport 2.x] [CI] allow for manual triggered cypress test jobs to accept inputs #5140

Merged

This was referenced Sep 28, 2023

[CI] Cypress test run spec input #5142

Merged

Cypress tests for data explorer opensearch-project/opensearch-dashboards-functional-test#862

Merged

abbyhu2000 pushed a commit that referenced this pull request Oct 2, 2023

[CI] accept inputs for manually triggered cypress tests (#5134) (#5140)

8a3e2ad

kavilla mentioned this pull request Oct 4, 2023

[CI][Cypress][Bug] use default SOURCE inputs for checkout #5197

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] allow for manual triggered cypress test jobs to accept inputs #5134

[CI] allow for manual triggered cypress test jobs to accept inputs #5134

kavilla commented Sep 27, 2023 •

edited

Loading

codecov bot commented Sep 27, 2023 •

edited

Loading

AMoo-Miki Sep 27, 2023 •

edited

Loading

kavilla Sep 27, 2023

joshuarrrr left a comment

joshuarrrr commented Sep 27, 2023

kavilla commented Sep 27, 2023

kavilla commented Sep 27, 2023 •

edited

Loading

github-actions bot commented Sep 28, 2023

[CI] allow for manual triggered cypress test jobs to accept inputs #5134

[CI] allow for manual triggered cypress test jobs to accept inputs #5134

Conversation

kavilla commented Sep 27, 2023 • edited Loading

Description

Issues Related

Screenshot

Testing the changes

Check List

codecov bot commented Sep 27, 2023 • edited Loading

Codecov Report

AMoo-Miki Sep 27, 2023 • edited Loading

Choose a reason for hiding this comment

kavilla Sep 27, 2023

Choose a reason for hiding this comment

joshuarrrr left a comment

Choose a reason for hiding this comment

joshuarrrr commented Sep 27, 2023

kavilla commented Sep 27, 2023

kavilla commented Sep 27, 2023 • edited Loading

github-actions bot commented Sep 28, 2023

[MANUAL CYPRESS TEST RUN RESULTS]

❌ Cypress test run failed!

Inputs:

Link to results:

kavilla commented Sep 27, 2023 •

edited

Loading

codecov bot commented Sep 27, 2023 •

edited

Loading

AMoo-Miki Sep 27, 2023 •

edited

Loading

kavilla commented Sep 27, 2023 •

edited

Loading