Reorganize GitHub workflows. #837

Yury-Fridlyand · 2022-09-16T21:18:41Z

Signed-off-by: Yury-Fridlyand [email protected]

Description

Remove obsolete workflows, mostly related to releasing ODFE artifacts.
Update actions in rest workflows.
Limit workflow scopes:
- Reduce file scope, for example, JDBC workflow will be triggered on JDBC changes only.
- Reduce action scope, so workflows shouldn't be triggered twice on PR.
Strictly define env version to avoid unexpected failures.
Upload test reports for SQL plugin workflow.
Activate tests for SQL CLI.
New workflow added which runs all tests for SQL plugin. Despite of the basic SQL workflow, it doesn't stop on the first failure and runs rests tests too. For more details see Add new test workflow. Bit-Quill/opensearch-project-sql#119.
New workflow would run on demand only. The button would appear after merge only.

Issues Resolved

JDBC/ODBC workflow won't be triggered on SQL plugin fixes and updates.
Workflows shouldn't be triggered twice in PRs anymore.
Cleanup.
Activated CodeQL workflow.
Activated SQL CLI testing.

Not included changes

ODBC workflow still uses MacOS env v.10.15 which is obsolete. It fails on MacOS 11. We can't switch to newer MacOS version without changing/updating ODBC build process. This will be done in scope of another PR.

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

**Reorganize GitHub workflows:** * Remove obsoleve workflows, mostly related to releaseing ODFE artifacts. * Update actions in rest workflows. * Limit workflow scopes: * Reduce file scope, for example, JDBC workflow will be triggered on JDBC changes only. * Reduce action scope, so workflows shouldn't be triggered twice on PR. * Strictly define env version to avoid unexpected failures. * Upload test reports for SQL plugin workflow. * Activate tests for SQL CLI. Signed-off-by: Yury-Fridlyand <[email protected]>

* Add new test workflow. Signed-off-by: Yury-Fridlyand <[email protected]>

.github/workflows/sql-test-workflow.yml

penghuo · 2022-09-27T16:06:51Z

.github/workflows/backport.yml

@@ -7,7 +7,7 @@ on:

 jobs:
  backport:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-20.04


I am not sure about that, do we have any known issue regarding to ubuntu-latest. Any reason change to 20.04?
I think opensearch-project use ubuntu-latest by default.

Please, follow these 2 links or see TLDR section
Bit-Quill#118 (comment)
https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
TL;DR
ubuntu-latest = ubuntu-20.04 now, but subject to change

penghuo · 2022-09-27T16:11:00Z

.github/workflows/sql-cli-test-and-build-workflow.yml

+        # tests are designed to run against http://localhost:9200, so we have to disable/remove security plugin
+      - name: Download and run OpenSearch
+        run: |
+          docker run -p 9200:9200 -e "discovery.type=single-node" -e "DISABLE_SECURITY_PLUGIN=true" --name test -d opensearchproject/opensearch:${{ matrix.opensearch-version }}


it may not work. e.g, currently, SQL plugin depend on OpenSearch 2.4. But OpenSearch 2.4 Docker Images is not released yet.

Currently, there are only 2.2.1 and latest in the test matrix. latest grabs the latest released version.
I can create another sub-job to download and build OpenSearch. So tests would run on the current version, e.g. 2.4 for now.

Might be worthwhile the next time someone has a chance.
Another side-effect, is that we would have an artefact to download for our own local development usage... instead of having to download and setup the latest OpenSearch.

penghuo · 2022-09-27T16:14:39Z

.github/workflows/sql-test-and-build-workflow.yml

      with:
        name: opensearch-sql
        path: opensearch-sql-builds
+
+    - name: Upload test reports


why add this? upload to where?

It create a downloadable artifact, like you see there on bottom by following the link. Default lifetime of those is few days or one week.
https://github.com/Bit-Quill/opensearch-project-sql/actions/runs/3087022572

joshuali925

a few questions

.github/workflows/draft-release-notes-workflow.yml

.github/workflows/link-checker.yml

.github/workflows/sql-cli-test-and-build-workflow.yml

.github/workflows/sql-test-workflow.yml

.github/workflows/sql-workbench-test-and-build-workflow.yml

Signed-off-by: Yury-Fridlyand <[email protected]>

joshuali925

thanks

acarbonetto · 2022-09-28T16:08:07Z

sql-cli/tests/test_opensearch_connection.py

@@ -39,6 +39,7 @@ def test_query(self, connection):
        )

    @estest
+    @pytest.mark.skip(reason="Test is not compatible with OpenSearch >= 2.3.0, it returns HTTP/503 instead of HTTP/400")


I've left a comment on the bug to revert this if it's fixed: #851

opensearch-trigger-bot · 2022-09-28T18:14:09Z

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.x 1.x
# Navigate to the new working tree
cd .worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-837-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 10e44ee6439cf130fdc9bff3081503951c42a1cd
# Push it to GitHub
git push --set-upstream origin backport/backport-837-to-1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-837-to-1.x.

opensearch-trigger-bot · 2022-09-28T18:14:10Z

The backport to main failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-main main
# Navigate to the new working tree
cd .worktrees/backport-main
# Create a new branch
git switch --create backport/backport-837-to-main
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 10e44ee6439cf130fdc9bff3081503951c42a1cd
# Push it to GitHub
git push --set-upstream origin backport/backport-837-to-main
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-main

Then, create a pull request where the base branch is main and the compare/head branch is backport/backport-837-to-main.

…nize-workflows Reorganize GitHub workflows. (cherry picked from commit 10e44ee)

…nize-workflows Reorganize GitHub workflows. (cherry picked from commit 10e44ee) Signed-off-by: Yury-Fridlyand <[email protected]>

Reorganize GitHub workflows. (cherry picked from commit 10e44ee) Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand requested a review from a team as a code owner September 16, 2022 21:18

Yury-Fridlyand marked this pull request as draft September 18, 2022 15:23

Yury-Fridlyand mentioned this pull request Sep 20, 2022

SQL ingestion and ops - standup #791

Open

Add new test workflow. (#119)

921791b

* Add new test workflow. Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand marked this pull request as ready for review September 23, 2022 16:50

MaxKsyunz previously approved these changes Sep 24, 2022

View reviewed changes

MaxKsyunz reviewed Sep 24, 2022

View reviewed changes

.github/workflows/sql-test-workflow.yml Show resolved Hide resolved

penghuo reviewed Sep 27, 2022

View reviewed changes

joshuali925 reviewed Sep 27, 2022

View reviewed changes

dai-chen added the infrastructure Changes to infrastructure, testing, CI/CD, pipelines, etc. label Sep 27, 2022

Typo fix.

489f2ad

Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand dismissed MaxKsyunz’s stale review via 489f2ad September 27, 2022 18:38

MaxKsyunz self-requested a review September 27, 2022 18:47

MaxKsyunz previously approved these changes Sep 27, 2022

View reviewed changes

Address PR feedback.

63c5137

Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand dismissed MaxKsyunz’s stale review via 63c5137 September 27, 2022 20:52

Yury-Fridlyand requested review from MaxKsyunz, joshuali925 and penghuo September 27, 2022 22:31

joshuali925 approved these changes Sep 27, 2022

View reviewed changes

penghuo approved these changes Sep 28, 2022

View reviewed changes

acarbonetto reviewed Sep 28, 2022

View reviewed changes

acarbonetto approved these changes Sep 28, 2022

View reviewed changes

Yury-Fridlyand added backport 1.x backport main labels Sep 28, 2022

Yury-Fridlyand merged commit 10e44ee into opensearch-project:2.x Sep 28, 2022

Yury-Fridlyand deleted the integ-reorganize-workflows branch September 28, 2022 18:13

Yury-Fridlyand added a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Sep 28, 2022

Merge pull request opensearch-project#837 from Bit-Quill/integ-reorga…

8fef49a

…nize-workflows Reorganize GitHub workflows. (cherry picked from commit 10e44ee)

Yury-Fridlyand mentioned this pull request Sep 28, 2022

[Backport 1.x] Reorganize GitHub workflows. #863

Merged

Yury-Fridlyand mentioned this pull request Sep 28, 2022

[Backport main] Reorganize GitHub workflows. #864

Merged

dai-chen pushed a commit that referenced this pull request Sep 28, 2022

Merge pull request #837 from Bit-Quill/integ-reorganize-workflows (#864)

9e28601

Reorganize GitHub workflows. (cherry picked from commit 10e44ee) Signed-off-by: Yury-Fridlyand <[email protected]>

dai-chen pushed a commit that referenced this pull request Sep 28, 2022

Merge pull request #837 from Bit-Quill/integ-reorganize-workflows (#863)

4a15c2f

Reorganize GitHub workflows. (cherry picked from commit 10e44ee) Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand mentioned this pull request Oct 12, 2022

Fix manual CI workflow and add name option. #904

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reorganize GitHub workflows. #837

Reorganize GitHub workflows. #837

Yury-Fridlyand commented Sep 16, 2022 •

edited

Loading

penghuo Sep 27, 2022

Yury-Fridlyand Sep 27, 2022 •

edited

Loading

penghuo Sep 27, 2022

Yury-Fridlyand Sep 27, 2022 •

edited

Loading

acarbonetto Sep 28, 2022

penghuo Sep 27, 2022

Yury-Fridlyand Sep 27, 2022 •

edited

Loading

joshuali925 left a comment

joshuali925 left a comment

acarbonetto Sep 28, 2022

opensearch-trigger-bot bot commented Sep 28, 2022

opensearch-trigger-bot bot commented Sep 28, 2022

Reorganize GitHub workflows. #837

Reorganize GitHub workflows. #837

Conversation

Yury-Fridlyand commented Sep 16, 2022 • edited Loading

Description

Issues Resolved

Not included changes

Check List

penghuo Sep 27, 2022

Choose a reason for hiding this comment

Yury-Fridlyand Sep 27, 2022 • edited Loading

Choose a reason for hiding this comment

penghuo Sep 27, 2022

Choose a reason for hiding this comment

Yury-Fridlyand Sep 27, 2022 • edited Loading

Choose a reason for hiding this comment

acarbonetto Sep 28, 2022

Choose a reason for hiding this comment

penghuo Sep 27, 2022

Choose a reason for hiding this comment

Yury-Fridlyand Sep 27, 2022 • edited Loading

Choose a reason for hiding this comment

joshuali925 left a comment

Choose a reason for hiding this comment

joshuali925 left a comment

Choose a reason for hiding this comment

acarbonetto Sep 28, 2022

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Sep 28, 2022

opensearch-trigger-bot bot commented Sep 28, 2022

Yury-Fridlyand commented Sep 16, 2022 •

edited

Loading

Yury-Fridlyand Sep 27, 2022 •

edited

Loading

Yury-Fridlyand Sep 27, 2022 •

edited

Loading

Yury-Fridlyand Sep 27, 2022 •

edited

Loading