Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/1.3' into 1.x (#2337)
Browse files Browse the repository at this point in the history
Merging all updates and improvements from 1.3 into 1.x so its build ecosystem is as robust as 1.3.

Signed-off-by: Peter Nied <[email protected]>
  • Loading branch information
peternied authored Jan 11, 2023
1 parent cb81cc6 commit 6ad2a18
Show file tree
Hide file tree
Showing 149 changed files with 3,241 additions and 916 deletions.
118 changes: 118 additions & 0 deletions .github/actions/start-opensearch-with-one-plugin/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
name: 'Launch OpenSearch with a single plugin installed'
description: 'Downloads latest build of OpenSearch, installs a plugin, executes a script and then starts OpenSearch on localhost:9200'

inputs:
opensearch-version:
description: 'The version of OpenSearch that should be used, e.g "3.0.0"'
required: true

plugin-name:
description: 'The name of the plugin to use, such as opensearch-security'
required: true

setup-script-name:
description: 'The name of the setup script you want to run i.e. "setup" (do not include file extension). Leave empty to indicate one should not be run.'
required: false

runs:
using: "composite"
steps:

# Configure longpath names if on Windows
- name: Enable Longpaths if on Windows
if: ${{ runner.os == 'Windows' }}
run: git config --system core.longpaths true
shell: pwsh

# Download OpenSearch
- name: Download OpenSearch for Windows
uses: peternied/download-file@v2
if: ${{ runner.os == 'Windows' }}
with:
url: https://artifacts.opensearch.org/snapshots/core/opensearch/${{ inputs.opensearch-version }}-SNAPSHOT/opensearch-min-${{ inputs.opensearch-version }}-SNAPSHOT-windows-x64-latest.zip

- name: Download OpenSearch for Linux
uses: peternied/download-file@v2
if: ${{ runner.os == 'Linux' }}
with:
url: https://artifacts.opensearch.org/snapshots/core/opensearch/${{ inputs.opensearch-version }}-SNAPSHOT/opensearch-min-${{ inputs.opensearch-version }}-SNAPSHOT-linux-x64-latest.tar.gz

# Extract downloaded zip
- name: Extract downloaded tar
if: ${{ runner.os == 'Linux' }}
run: |
tar -xzf opensearch-*.tar.gz
rm -f opensearch-*.tar.gz
shell: bash

- name: Extract downloaded zip
if: ${{ runner.os == 'Windows' }}
run: |
tar -xzf opensearch-min-${{ inputs.opensearch-version }}-SNAPSHOT-windows-x64-latest.zip
del opensearch-min-${{ inputs.opensearch-version }}-SNAPSHOT-windows-x64-latest.zip
shell: pwsh

# Install the plugin
- name: Install Plugin into OpenSearch for Linux
if: ${{ runner.os == 'Linux'}}
run: |
chmod +x ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT/bin/opensearch-plugin
/bin/bash -c "yes | ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT/bin/opensearch-plugin install file:$(pwd)/opensearch-security.zip"
shell: bash

- name: Install Plugin into OpenSearch for Windows
if: ${{ runner.os == 'Windows'}}
run: |
'y' | .\opensearch-${{ inputs.opensearch-version }}-SNAPSHOT\bin\opensearch-plugin.bat install file:$(pwd)\${{ inputs.plugin-name }}.zip
shell: pwsh

# Run any configuration scripts
- name: Run Setup Script for Linux
if: ${{ runner.os == 'Linux' && inputs.setup-script-name != '' }}
run: |
echo "running linux setup"
chmod +x ./${{ inputs.setup-script-name }}.sh
./${{ inputs.setup-script-name }}.sh
shell: bash

- name: Run Setup Script for Windows
if: ${{ runner.os == 'Windows' && inputs.setup-script-name != '' }}
run: .\${{ inputs.setup-script-name }}.bat
shell: pwsh

# Run OpenSearch
- name: Run OpenSearch with plugin on Linux
if: ${{ runner.os == 'Linux'}}
run: /bin/bash -c "./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT/bin/opensearch &"
shell: bash

- name: Run OpenSearch with plugin on Windows
if: ${{ runner.os == 'Windows'}}
run: start .\opensearch-${{ inputs.opensearch-version }}-SNAPSHOT\bin\opensearch.bat
shell: pwsh

# Give the OpenSearch process some time to boot up before sending any requires, might need to increase the default time!
- name: Sleep while OpenSearch starts
uses: peternied/action-sleep@v1
with:
seconds: 30

# Verify that the server is operational
- name: Check OpenSearch Running on Linux
if: ${{ runner.os != 'Windows'}}
run: curl https://localhost:9200/_cat/plugins -u 'admin:admin' -k -v
shell: bash

- name: Check OpenSearch Running on Windows
if: ${{ runner.os == 'Windows'}}
run: |
$credentialBytes = [Text.Encoding]::ASCII.GetBytes("admin:admin")
$encodedCredentials = [Convert]::ToBase64String($credentialBytes)
$baseCredentials = "Basic $encodedCredentials"
$Headers = @{ Authorization = $baseCredentials }
Invoke-WebRequest -SkipCertificateCheck -Uri 'https://localhost:9200/_cat/plugins' -Headers $Headers;
shell: pwsh

- if: always()
run: cat ./opensearch-${{ inputs.opensearch-version }}-SNAPSHOT/logs/opensearch.log
shell: bash
108 changes: 82 additions & 26 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,52 +8,108 @@ env:
jobs:
build:
name: build
runs-on: ubuntu-latest
strategy:
fail-fast: true
fail-fast: false
matrix:
jdk: [8, 11, 14]
platform: ["ubuntu-latest", "windows-latest"]
runs-on: ${{ matrix.platform }}

steps:

- name: Set up JDK
- name: Set up JDK for build and test on 8 and 11
if: matrix.jdk != '14'
uses: actions/setup-java@v2
with:
distribution: temurin # Temurin is a distribution of adoptium
java-version: ${{ matrix.jdk }}

- name: Set up JDK for build and test on 14
if: matrix.jdk == '14'
uses: actions/setup-java@v1
with:
distribution: temurin
java-version: ${{ matrix.jdk }}

- name: Checkout security
uses: actions/checkout@v2

- name: Initialize CodeQL
uses: github/codeql-action/init@v1
- name: Build and Test
uses: gradle/gradle-build-action@v2
with:
languages: java
arguments: |
build test -Dbuild.snapshot=false
-x checkstyleMain
-x checkstyleTest
- name: Cache Gradle packages
uses: actions/cache@v2
- name: Coverage
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./build/reports/jacoco/test/jacocoTestReport.xml

- uses: actions/upload-artifact@v3
if: always()
with:
name: ${{ matrix.platform }}-JDK${{ matrix.jdk }}-reports
path: |
~/.gradle/caches
~/.gradle/wrapper
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
restore-keys: |
${{ runner.os }}-gradle-
./build/reports/
- name: check archive for debugging
if: always()
run: echo "Check the artifact ${{ matrix.platform }}-JDK${{ matrix.jdk }}-reports for detailed test results"

- name: Checkstyle
run: ./gradlew clean checkstyleMain checkstyleTest
code-ql:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v1
with:
java-version: 11
- uses: github/codeql-action/init@v1
with:
languages: java
- run: ./gradlew clean build -Dbuild.snapshot=false -x test
- uses: github/codeql-action/analyze@v1

- name: Package
run: ./gradlew clean build -Dbuild.snapshot=false -x test
build-artifact-names:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v1
- uses: actions/setup-java@v1
with:
java-version: 11

- name: Test
run: OPENDISTRO_SECURITY_TEST_OPENSSL_OPT=true ./gradlew test -i
- run: |
security_plugin_version=$(./gradlew properties -q | grep -E '^version:' | awk '{print $2}')
security_plugin_version_no_snapshot=$(echo $security_plugin_version | sed 's/-SNAPSHOT//g')
security_plugin_version_only_number=$(echo $security_plugin_version_no_snapshot | cut -d- -f1)
test_qualifier=alpha2
echo "SECURITY_PLUGIN_VERSION=$security_plugin_version" >> $GITHUB_ENV
echo "SECURITY_PLUGIN_VERSION_NO_SNAPSHOT=$security_plugin_version_no_snapshot" >> $GITHUB_ENV
echo "SECURITY_PLUGIN_VERSION_ONLY_NUMBER=$security_plugin_version_only_number" >> $GITHUB_ENV
echo "TEST_QUALIFIER=$test_qualifier" >> $GITHUB_ENV
- run: |
echo ${{ env.SECURITY_PLUGIN_VERSION }}
echo ${{ env.SECURITY_PLUGIN_VERSION_NO_SNAPSHOT }}
echo ${{ env.SECURITY_PLUGIN_VERSION_ONLY_NUMBER }}
echo ${{ env.TEST_QUALIFIER }}
- run: ./gradlew clean assemble && test -s ./build/opensearch-security-${{ env.SECURITY_PLUGIN_VERSION }}.jar

- name: Coverage
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./build/jacoco/test/jacocoTestReport.xml
- run: ./gradlew clean assemble -Dbuild.snapshot=false && test -s ./build/opensearch-security-${{ env.SECURITY_PLUGIN_VERSION_NO_SNAPSHOT }}.jar

- run: ./gradlew clean assemble -Dbuild.snapshot=false -Dbuild.version_qualifier=${{ env.TEST_QUALIFIER }} && test -s ./build/opensearch-security-${{ env.SECURITY_PLUGIN_VERSION_ONLY_NUMBER }}-${{ env.TEST_QUALIFIER }}.jar

- run: ./gradlew clean assemble -Dbuild.version_qualifier=${{ env.TEST_QUALIFIER }} && test -s ./build/opensearch-security-${{ env.SECURITY_PLUGIN_VERSION_ONLY_NUMBER }}-${{ env.TEST_QUALIFIER }}-SNAPSHOT.jar

- run: |
## EXISTING_OS_VERSION outputs the major version, example as 2
EXISTING_OS_VERSION=$(./gradlew properties | grep opensearch.version | cut -d':' -f2- | awk '{$1=$1};1' | cut -d '-' -f1 | cut -d '.' -f1)
## INCREMENT_OS_VERSION in an increment of 1, example if EXISTING_OS_VERSION is 2, INCREMENT_OS_VERSION is 3
INCREMENT_OS_VERSION=$((++EXISTING_OS_VERSION))
./gradlew clean updateVersion -DnewVersion=$INCREMENT_OS_VERSION.0.0-SNAPSHOT
test `./gradlew properties | grep opensearch.version | cut -d':' -f2- | awk '{$1=$1};1'` = $INCREMENT_OS_VERSION.0.0-SNAPSHOT
- name: List files in the build directory if there was an error
run: ls -al ./build/
if: failure()
56 changes: 56 additions & 0 deletions .github/workflows/plugin_install.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: Plugin Install

on: [push, pull_request, workflow_dispatch]

env:
OPENSEARCH_VERSION: 1.4.0
PLUGIN_NAME: opensearch-security

jobs:
plugin-install:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
jdk: [8, 11, 14]
runs-on: ${{ matrix.os }}

steps:
- name: Set up JDK
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.jdk }}

- name: Checkout Branch
uses: actions/checkout@v2

- name: Assemble target plugin
uses: gradle/gradle-build-action@v2
with:
arguments: assemble

# Move and rename the plugin for installation
- name: Move and rename the plugin for installation
run: mv ./build/distributions/${{ env.PLUGIN_NAME }}-*-SNAPSHOT.zip ${{ env.PLUGIN_NAME }}.zip
shell: bash

- name: Create Setup Script
if: ${{ runner.os == 'Linux' }}
run: |
cat > setup.sh <<'EOF'
chmod +x ./opensearch-${{ env.OPENSEARCH_VERSION }}-SNAPSHOT/plugins/${{ env.PLUGIN_NAME }}/tools/install_demo_configuration.sh
/bin/bash -c "./opensearch-${{ env.OPENSEARCH_VERSION }}-SNAPSHOT/plugins/${{ env.PLUGIN_NAME }}/tools/install_demo_configuration.sh -y"
EOF
- name: Create Setup Script
if: ${{ runner.os == 'Windows' }}
run: |
New-Item .\setup.bat -type file
Set-Content .\setup.bat -Value "powershell.exe .\opensearch-${{ env.OPENSEARCH_VERSION }}-SNAPSHOT\plugins\${{ env.PLUGIN_NAME }}\tools\install_demo_configuration.bat -y"
- name: Run Opensearch with A Single Plugin
uses: ./.github/actions/start-opensearch-with-one-plugin
with:
opensearch-version: ${{ env.OPENSEARCH_VERSION }}
plugin-name: ${{ env.PLUGIN_NAME }}
setup-script-name: setup
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## Contributing to this Project

OpenSearch is a community project that is built and maintained by people just like **you**.
[This document](https://github.com/opensearch-project/.github/blob/main/CONTRIBUTING.md) explains how you can contribute to this and related projects.
[This document](https://github.com/opensearch-project/.github/blob/main/CONTRIBUTING.md) explains how you can contribute to this and related projects.
Loading

0 comments on commit 6ad2a18

Please sign in to comment.