Build OpenSearch on Windows (#767)

* Refactored TemporaryFile and GitRepository into context managers. Signed-off-by: dblock <[email protected]> * Remove mkdtemp. Signed-off-by: dblock <[email protected]> * Fix tests on Windows. Signed-off-by: dblock <[email protected]> * Execute commands using bash. Signed-off-by: dblock <[email protected]> * Support windows zip for OpenSearch-min. Signed-off-by: dblock <[email protected]>
opensearch-project · Oct 22, 2021 · 7f3e41f · 7f3e41f
1 parent 6ae52ca
commit 7f3e41f
Show file tree

Hide file tree

Showing 64 changed files with 444 additions and 272 deletions.
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -17,6 +17,7 @@ jobs:
         os:
           - ubuntu-latest
           - macos-latest
+          - windows-latest
     runs-on: ${{ matrix.os }}
     env:
       PYTHON_VERSION: 3.7

diff --git a/.gitignore b/.gitignore
@@ -16,3 +16,5 @@ out.txt
 
 /artifacts/
 /bundle/
+
+/.vscode/
diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md
@@ -24,7 +24,7 @@ Fork this repository on GitHub, and clone locally with `git clone`.
 
 #### Pyenv
 
-Use pyenv to manage multiple versions of Python. This can be installed with [pyenv-installer](https://github.com/pyenv/pyenv-installer).
+Use pyenv to manage multiple versions of Python. This can be installed with [pyenv-installer](https://github.com/pyenv/pyenv-installer) on Linux and MacOS, and [pyenv-win](https://github.com/pyenv-win/pyenv-win#installation) on Windows.
 
 ```
 curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
@@ -42,7 +42,7 @@ Python 3.7.11
 If you are using pyenv.
 
 ```
-pyenv install 3.7.12
+pyenv install 3.7.12 # use 3.7.9 on Windows, the latest at the time of writing this
 pyenv global 3.7.12
 ```
 
@@ -57,7 +57,10 @@ $ pipenv --version
 pipenv, version 19.0
 ```
 
+On Windows, run `pyenv rehash` if `pipenv` cannot be found. This rehashes pyenv shims, creating a `pipenv` file in `/.pyenv/pyenv-win/shims/`.
+
 #### NVM and Node
+
 Install [nvm](https://github.com/nvm-sh/nvm/blob/master/README.md) to use the Node 10.24.1 version as it is required
 
 ```
@@ -66,6 +69,7 @@ nvm install v10.24.1
 ```
 
 #### Yarn
+
 [Yarn](https://classic.yarnpkg.com/en/docs/install) is required for building and running the OpenSearch Dashboards and plugins
 
 ```

diff --git a/Pipfile.lock b/Pipfile.lock
diff --git a/run.sh b/run.sh
@@ -20,4 +20,4 @@ export PIPENV_PIPFILE="$DIR/Pipfile"
 python3 -m pipenv install
 
 echo "Running "$1" ${@:2} ..."
-python3 -m pipenv run "$1" ${@:2}
+python3 -m pipenv run python "$1" ${@:2}
diff --git a/scripts/components/OpenSearch/build.sh b/scripts/components/OpenSearch/build.sh
@@ -72,18 +72,42 @@ mkdir -p $OUTPUT/maven/org/opensearch
 # Copy maven publications to be promoted
 cp -r ./build/local-test-repo/org/opensearch "${OUTPUT}"/maven/org
 
+# Assemble distribution artifact
+# see https://github.com/opensearch-project/OpenSearch/blob/main/settings.gradle#L34 for other distribution targets
+
 [ -z "$PLATFORM" ] && PLATFORM=`uname -s` | awk '{print tolower($0)}'
 [ -z "$ARCHITECTURE" ] && ARCHITECTURE=`uname -m`
 
-# Assemble distribution artifact
-# see https://github.com/opensearch-project/OpenSearch/blob/main/settings.gradle#L34 for other distribution targets
+case "$(uname -s)" in
+    Linux*) 
+        PACKAGE="tar"
+        EXT="tar.gz"
+        ;;
+    Darwin*)
+        PACKAGE="tar"
+        EXT="tar.gz"
+        ;;
+    CYGWIN*)
+        PACKAGE="zip"
+        EXT="zip"
+        ;;
+    MINGW*)
+        PACKAGE="zip"
+        EXT="zip"
+        ;;
+    *)
+        echo "Unsupported system: $(uname -s)"
+        exit 1
+        ;;
+esac
+
 case $ARCHITECTURE in
     x64)
-        TARGET="$PLATFORM-tar"
+        TARGET="$PLATFORM-$PACKAGE"
         QUALIFIER="$PLATFORM-x64"
         ;;
     arm64)
-        TARGET="$PLATFORM-arm64-tar"
+        TARGET="$PLATFORM-arm64-$PACKAGE"
         QUALIFIER="$PLATFORM-arm64"
         ;;
     *)
@@ -96,7 +120,7 @@ esac
 
 # Copy artifact to dist folder in bundle build output
 [[ "$SNAPSHOT" == "true" ]] && IDENTIFIER="-SNAPSHOT"
-ARTIFACT_BUILD_NAME=`ls distribution/archives/$TARGET/build/distributions/ | grep "opensearch-min.*$QUALIFIER.tar.gz"`
+ARTIFACT_BUILD_NAME=`ls distribution/archives/$TARGET/build/distributions/ | grep "opensearch-min.*$QUALIFIER.$EXT"`
 mkdir -p "${OUTPUT}/dist"
 cp distribution/archives/$TARGET/build/distributions/$ARTIFACT_BUILD_NAME "${OUTPUT}"/dist/$ARTIFACT_BUILD_NAME
 

diff --git a/src/assemble_workflow/bundle_opensearch.py b/src/assemble_workflow/bundle_opensearch.py
@@ -12,6 +12,6 @@
 class BundleOpenSearch(Bundle):
     def install_plugin(self, plugin):
         tmp_path = self._copy_component(plugin, "plugins")
-        cli_path = os.path.join(self.archive_path, "bin/opensearch-plugin")
+        cli_path = os.path.join(self.archive_path, "bin", "opensearch-plugin")
         self._execute(f"{cli_path} install --batch file:{tmp_path}")
         super().install_plugin(plugin)
Original file line number	Diff line number	Diff line change
Expand Up		@@ -16,3 +16,5 @@ out.txt

		/artifacts/
		/bundle/

		/.vscode/