Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[META] CI Improvement Plan #4019

Open
seanneumann opened this issue May 12, 2023 · 1 comment
Open

[META] CI Improvement Plan #4019

seanneumann opened this issue May 12, 2023 · 1 comment
Assignees
Labels
ci enhancement New feature or request

Comments

@seanneumann
Copy link
Contributor

seanneumann commented May 12, 2023

We'd like to improve our CI for OpenSearch Dashboards. Creating this issue to start forming the plan.

Initial AI: baseline metrics to measure progress against and establish goal.

CI Labelled Issues

@kavilla
Copy link
Member

kavilla commented Sep 1, 2023

Device matrix testing #4803

kavilla added a commit to kavilla/OpenSearch-Dashboards-1 that referenced this issue Sep 27, 2023
Utilizing the GitHub `workflow_dispatch`, we can trigger
a job with a specific inputs.

By default, we run from this repo:
https://github.com/opensearch-project/opensearch-dashboards-functional-test

This workflow uses the target branch of the PR to pull down
from the FTRepo and run the tests from that specific branch.

For example, PRs against `main` will pull down `main` from
the FTRepo.

The problem occurs when new functionality is opened or a bug is
fixed and, per industry standard, we want to see tests added
to ensure functionality, stability, and raise the bar.

However, the cypress tests PR into the FTRepo depends on the new code
to be merged for the CI within the FTRepo to work. So we have a
stalemate that usually slows down PR review time.

Here, a manual run can be triggered by a maintainer. It will utilize
the source branch from the PR if provided.

Related to:
opensearch-project#4019

Signed-off-by: Kawika Avilla <[email protected]>
kavilla added a commit that referenced this issue Sep 27, 2023
Utilizing the GitHub `workflow_dispatch`, we can trigger
a job with a specific inputs.

By default, we run from this repo:
https://github.com/opensearch-project/opensearch-dashboards-functional-test

This workflow uses the target branch of the PR to pull down
from the FTRepo and run the tests from that specific branch.

For example, PRs against `main` will pull down `main` from
the FTRepo.

The problem occurs when new functionality is opened or a bug is
fixed and, per industry standard, we want to see tests added
to ensure functionality, stability, and raise the bar.

However, the cypress tests PR into the FTRepo depends on the new code
to be merged for the CI within the FTRepo to work. So we have a
stalemate that usually slows down PR review time.

Here, a manual run can be triggered by a maintainer. It will utilize
the source branch from the PR if provided.

Related to:
#4019

Signed-off-by: Kawika Avilla <[email protected]>
opensearch-trigger-bot bot pushed a commit that referenced this issue Sep 27, 2023
Utilizing the GitHub `workflow_dispatch`, we can trigger
a job with a specific inputs.

By default, we run from this repo:
https://github.com/opensearch-project/opensearch-dashboards-functional-test

This workflow uses the target branch of the PR to pull down
from the FTRepo and run the tests from that specific branch.

For example, PRs against `main` will pull down `main` from
the FTRepo.

The problem occurs when new functionality is opened or a bug is
fixed and, per industry standard, we want to see tests added
to ensure functionality, stability, and raise the bar.

However, the cypress tests PR into the FTRepo depends on the new code
to be merged for the CI within the FTRepo to work. So we have a
stalemate that usually slows down PR review time.

Here, a manual run can be triggered by a maintainer. It will utilize
the source branch from the PR if provided.

Related to:
#4019

Signed-off-by: Kawika Avilla <[email protected]>
(cherry picked from commit 34a8594)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

# Conflicts:
#	CHANGELOG.md
kavilla added a commit to kavilla/OpenSearch-Dashboards-1 that referenced this issue Sep 28, 2023
Originally:
opensearch-project#5134

I believed it to be a good idea to ensure that triggering the CI
should always run a batch of tests. But I realized that some tests
can be added and just be bad tests until updated. Or if we want
to verify a flaky test in the CI.

So instead of only making it additional tests, the spec input will
complete replace the default tests (unless you append to the default).

Empty will run the default spec still. This will append on to the
PR comment with the spec that was run. So at least maintainers can
see it passed for a set of tests but it did not run the ones
we usually run.

Also adding some formatting for empty values for the PR comment.

Issue related:
opensearch-project#4019

Signed-off-by: Kawika Avilla <[email protected]>
kavilla added a commit that referenced this issue Sep 29, 2023
Originally:
#5134

I believed it to be a good idea to ensure that triggering the CI
should always run a batch of tests. But I realized that some tests
can be added and just be bad tests until updated. Or if we want
to verify a flaky test in the CI.

So instead of only making it additional tests, the spec input will
complete replace the default tests (unless you append to the default).

Empty will run the default spec still. This will append on to the
PR comment with the spec that was run. So at least maintainers can
see it passed for a set of tests but it did not run the ones
we usually run.

Also adding some formatting for empty values for the PR comment.

Issue related:
#4019

Signed-off-by: Kawika Avilla <[email protected]>
opensearch-trigger-bot bot pushed a commit that referenced this issue Oct 2, 2023
Originally:
#5134

I believed it to be a good idea to ensure that triggering the CI
should always run a batch of tests. But I realized that some tests
can be added and just be bad tests until updated. Or if we want
to verify a flaky test in the CI.

So instead of only making it additional tests, the spec input will
complete replace the default tests (unless you append to the default).

Empty will run the default spec still. This will append on to the
PR comment with the spec that was run. So at least maintainers can
see it passed for a set of tests but it did not run the ones
we usually run.

Also adding some formatting for empty values for the PR comment.

Issue related:
#4019

Signed-off-by: Kawika Avilla <[email protected]>
(cherry picked from commit 6154e2d)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
kavilla pushed a commit that referenced this issue Oct 2, 2023
Originally:
#5134

I believed it to be a good idea to ensure that triggering the CI
should always run a batch of tests. But I realized that some tests
can be added and just be bad tests until updated. Or if we want
to verify a flaky test in the CI.

So instead of only making it additional tests, the spec input will
complete replace the default tests (unless you append to the default).

Empty will run the default spec still. This will append on to the
PR comment with the spec that was run. So at least maintainers can
see it passed for a set of tests but it did not run the ones
we usually run.

Also adding some formatting for empty values for the PR comment.

Issue related:
#4019


(cherry picked from commit 6154e2d)

Signed-off-by: Kawika Avilla <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants