Refactor: Docker Compose Easy Setup (#250)

* Refactor: Docker Compose Easy Setup Signed-off-by: jay-dee7 <[email protected]> * Changed action to reflect the updates * Add: long-syntax for ports in docker-compose config Signed-off-by: jay-dee7 <[email protected]> * File structure and workflows * Switched permissions * Copy as root then own it * Added permissions * Created localnet setup extracted from workflow * Removed volume from Dockerfile * Switched directory to match changes * Fixed to match new image settings * External laddr availability * Switched test port to avoid clashes * Bypass defaults * Removed excess * Removed moniker name in container env vars Signed-off-by: jay-dee7 <[email protected]> * Fixed supervisor config to match dirs * Removed unwanted quotes * Switched to reflect main * Single Dockerfile for mainnet & testnet * Adjusted to old dir * Removed excess * Upgraded testnet version beforehand * Added whole home dir volume * Multiple driver options & fixes * Testnet ref typo fix * Consistent across mainnet & testnet * Moved environment variables to host * Move single image localnet to tests * Fix naming of build tools * Fx liinks in workflows * Move docker localnet to tests * Fix documentation * Fix case * Fix broken links * Allow 0, 403 status codes for link checker * Fix broken links * Fix file names * Fx links, dockerfile * Fix test pipeline * Fix test pipeline * Fix test * Revert volume names * Fix corner case where host ip is invalid at first launch * Optimize docker file * Corrected spelling in filename * Fix script name in the lint pipeline * Typo fixes in GitHub Actions release workflow * Cosmetic switch to put exposed ports in same order as in config files * Swapping parameter order around in node-runner.sh * Added explanatory comments to Docker Compose file * Added extended descriptions for inside-container configuration file for mainnet * Update Dockerfile * Modified configuration for volumes in Docker Compose file * Extended descriptions in host environment variables * Update .Dockerignore * Update release.yml * Update .gitignore * Changed node-runner script to copy genesis and seed config data from Docker Compose config locations * Optimise Dockerfile by removing unnecessary requirements - Removed unused apt-get install packages in Docker build stages - Added comments for readability * Fixed errors due to incorrect commenting style in Compose environment file * Fixed incorrect variable names in Docker Compose file * Updated Docker Compose files and moved directories * Changed host and container environment files * Update node-runner.sh * Bumped version to v0.3.5 * Updated Dockerfile release compose * Moved production build stage to separate file * Renaming host.env to docker-compose.env * Renaming node-runner.sh to entrypoint.sh * Added build stage to Docker Compose file * Created separate stage Dockerfile for cheqd-node release images * Fix container's parameters Signed-off-by: Andrew Nikitin <[email protected]> * Fix cheqd-node image building Signed-off-by: Andrew Nikitin <[email protected]> * Rename job name Signed-off-by: Andrew Nikitin <[email protected]> * Fix yaml errors Signed-off-by: Andrew Nikitin <[email protected]> * Change Dockerfile to multi-stage build * Made build environment variable optional * Commented out build section in Docker Compose file * Delete Dockerfile.release * Debugging docker-compose Co-authored-by: Andrew Nikitin <[email protected]> * Debugging docker-compose * Fix image building * Change entrypoint for cheqd_noded_docker cmd Signed-off-by: Andrew Nikitin <[email protected]> * Make 77 for all dirs Signed-off-by: Andrew Nikitin <[email protected]> * Add UID and GID for image buildingI Signed-off-by: Andrew Nikitin <[email protected]> * Change params passing for UID and GID Signed-off-by: Andrew Nikitin <[email protected]> * Debug Signed-off-by: Andrew Nikitin <[email protected]> * Change entrypoints for tests Signed-off-by: Andrew Nikitin <[email protected]> * Update Dockerfile * Update docker-compose.yml * Revert "Update docker-compose.yml" This reverts commit d727e9d. Co-authored-by: Tasos <[email protected]> Co-authored-by: Alexandr Kolesov <[email protected]> Co-authored-by: Ankur Banerjee <[email protected]> Co-authored-by: Andrew Nikitin <[email protected]> Signed-off-by: Andrew Nikitin <[email protected]>
cheqd · Feb 25, 2022 · b3a6eee · b3a6eee
1 parent 68ddd2b
commit b3a6eee
Show file tree

Hide file tree

Showing 39 changed files with 479 additions and 315 deletions.
diff --git a/.Dockerignore b/.Dockerignore
@@ -1,8 +1,25 @@
-ci
-.devcontainer
+### APP-SPECIFIC
 .github
+*.md
+build-tools/*
+docs/*
+architecture/*
+persistent_chains/*
+
+
+### GENERAL EXCLUSIONS ###
+
+### macOS ###
+# General
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# IDE Exclusions
+.devcontainer
+.vscode
+.vscode-upload.json
 .idea
-vue/node_modules
-vue/src/store/generated
-vue/dist
-secret.yml
+
+# Python
+.pytest_cache
diff --git a/.github/scripts/ensure_set_euox_pipefail.sh b/.github/scripts/ensure_set_euox_pipefail.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+# Ensures that all bash scripts in the repository use `set -euox pipefail` or `set -euo pipefail` statement at the beginning.
+
+set -euo pipefail
+
+INVALID_FILES_FOUND=0
+
+for BASH_SCRIPT in $(find . -type f -name "*.sh")
+do
+    if ( ! grep -q "set -euo pipefail" "${BASH_SCRIPT}" ) && ( ! grep -q "set -euox pipefail" "${BASH_SCRIPT}" )
+    then
+        echo "${BASH_SCRIPT}"
+        INVALID_FILES_FOUND=1
+    fi
+done
+
+if [[ INVALID_FILES_FOUND -eq 1 ]]
+then
+    echo ""
+    echo "The bash scripts above must include either 'set -euo pipefail' or 'set -euox pipefail."
+    exit 1
+fi
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
@@ -0,0 +1,63 @@
+name: Lint
+
+on: [push]
+
+jobs:
+
+  # We can't use VALIDATE_GO from super linter because of this issue:
+  # https://github.com/github/super-linter/issues/143
+  go-lint:
+    name: Lint go
+    runs-on: ubuntu-20.04
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: golangci-lint
+        uses: golangci/golangci-lint-action@v2
+        with:
+          version: v1.43.0
+          args: --timeout 5m0s
+
+  md-link-check:
+    name: Check markdown links
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Check if Markdown links are valid
+        uses: gaurav-nelson/github-action-markdown-link-check@v1
+        with: 
+          config-file: '.github/linters/mlc_config.json'
+          folder-path: "."
+
+  sh-euox-pipefail-check:
+    name: Check for 'set -euox pipefail' in shell scripts
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Check that all shell scripts use 'set -euox pipefail'
+        run: bash .github/scripts/ensure_set_euox_pipefail.sh
+
+  super-lint:
+    name: Run super linter
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    - name: Super-Linter
+      uses: github/super-linter/slim@v4
+      env:
+        VALIDATE_MARKDOWN: true
+        VALIDATE_XML: true
+        VALIDATE_YAML: true
+        VALIDATE_OPENAPI: true
+
+        VALIDATE_ALL_CODEBASE: true
+        LOG_LEVEL: WARN
+
+        DEFAULT_BRANCH: main
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/markdown-ci/mlc_config.json b/.github/workflows/markdown-ci/mlc_config.json
@@ -1,3 +1,3 @@
 {
-    "aliveStatusCodes": [200, 206, 999]
-}
+    "aliveStatusCodes": [0, 200, 206, 403, 999]
+}
diff --git a/.github/workflows/node-ci.yml b/.github/workflows/node-ci.yml
@@ -9,7 +9,7 @@ on:
     - '.gitbook/**'
 
 env:
-  NODE_CONFIGS_BASE: "/home/runner/work/cheqd-node/cheqd-node/tests/networks/docker_compose/node_configs"
+  NODE_CONFIGS_BASE: "/home/runner/work/cheqd-node/cheqd-node/tests/networks/docker-compose-localnet/node_configs"
   PACKAGE_NAME: "cheqd-node"
   VERSION: "1.0.0" # Package build scripts require version to be provided
 
@@ -105,25 +105,25 @@ jobs:
         run: sudo chmod +x /home/runner/go/bin/cheqd-noded
 
       - name: Make intermediate archive
-        working-directory: ./build_tools
+        working-directory: ./build-tools
         run: |
-          ./build_tar.sh ${{ env.PACKAGE_NAME }} ${{ env.VERSION }} 
+          ./build-tar.sh ${{ env.PACKAGE_NAME }} ${{ env.VERSION }} 
 
       - name: Install fpm
         run: |
           sudo apt-get install ruby ruby-dev rubygems build-essential
           sudo gem install --no-document fpm
 
       - name: Build deb # Depends on tar archive
+        working-directory: ./build-tools
         run: |
-          ./build_deb.sh ${{ env.PACKAGE_NAME }} ${{ env.VERSION }}
-        working-directory: ./build_tools
+          ./build-deb.sh ${{ env.PACKAGE_NAME }} ${{ env.VERSION }}
 
       - name: Store deb package artifact
         uses: actions/upload-artifact@v2
         with:
           name: ${{ env.PACKAGE_NAME }}_${{ env.VERSION }}_amd64.deb
-          path: build_tools/output/${{ env.PACKAGE_NAME }}_${{ env.VERSION }}_amd64.deb
+          path: build-tools/output/${{ env.PACKAGE_NAME }}_${{ env.VERSION }}_amd64.deb
 
   build-node-image:
     name: Build cheqd-node image
@@ -134,7 +134,10 @@ jobs:
           fetch-depth: 0 # FIXME: Starport requires full repository
 
       - name: Build
-        run: docker build -f docker/cheqd_node/Dockerfile --no-cache -t cheqd-node  --build-arg UID=$(id -u) --build-arg GID=$(id -g) . # TODO: Get rid of UID and GID
+        run: |
+          export CHEQD_NODE_VERSION=local
+          docker compose --env-file docker/persistent-chains/docker-compose.env -f docker/persistent-chains/docker-compose.yml build --build-arg UID=$(id -u) --build-arg GID=$(id -g)
+          docker tag ghcr.io/cheqd/cheqd-node:local cheqd-node
 
       - name: Save
         run: docker save -o cheqd-node-image.tar cheqd-node
@@ -162,7 +165,7 @@ jobs:
         run: docker load -i cheqd-node-image.tar
 
       - name: Build
-        run: docker build -f docker/single_image_testnet/Dockerfile --no-cache -t cheqd-testnet .
+        run: docker build -f tests/networks/docker-localnet/Dockerfile --no-cache -t cheqd-testnet .
 
       - name: Save
         run: docker save -o cheqd-testnet-image.tar cheqd-testnet
@@ -198,23 +201,23 @@ jobs:
         uses: actions/checkout@v2
 
       - name: Set up 4 validators + 2 observers node docker pool  # TODO: Replace with testnet image
-        working-directory: ./tests/networks/docker_compose
+        working-directory: ./tests/networks/docker-compose-localnet
         run: |
-          ./gen_node_configs.sh
-          ./run_docker.sh
+          ./gen-node-configs.sh
+          ./run-docker.sh
           ./wait.sh '[[ $(curl -s -N localhost:26657/block | jq -cr '"'"'.result.block.last_commit.height'"'"') -gt 1 ]] && echo "Height is more than 1"'
 
       - name: Add observer node
-        working-directory: ./tests/networks/docker_compose
+        working-directory: ./tests/networks/docker-compose-localnet
         run: |
-          ./add_observer.sh
+          ./add-observer.sh
           cheqd-noded status -n tcp://localhost:26677 2>&1
           ./wait.sh '[[ $(cheqd-noded status -n '"'"'tcp://localhost:26677'"'"' 2>&1 | wc -l) == 1 ]] && echo "New node returns status!"'
 
       - name: Promote observer to validator
-        working-directory: ./tests/networks/docker_compose
+        working-directory: ./tests/networks/docker-compose-localnet
         run: |
-          bash -x promote_validator.sh
+          bash -x promote-validator.sh
           cheqd-noded query staking validators --node "http://localhost:26657" | sed -nr 's/.*status: (.*?).*/\1/p' $1 | while read x; do [[ "BOND_STATUS_BONDED" == $x ]] && echo "Validator's status is bonded!" || exit 1 ; done
           ./wait.sh '[[ $(curl -s localhost:26657/block | sed -nr '"'"'s/.*signature": (.*?).*/\1/p'"'"' | wc -l) == 5 ]] && echo "There are 5 validators signatures in block!"'
           ./wait.sh '[[ $(curl -s localhost:26657/block | sed -nr '"'"'s/.*(signature": null).*/\1/p'"'"' | wc -l) == 0 ]] && echo "There are no null signatures in block!"'
@@ -254,10 +257,10 @@ jobs:
       - uses: actions/checkout@v2
 
       - name: Set up 4 validators + 2 observers node docker pool for local test run # Replace with testnet image
-        working-directory: ./tests/networks/docker_compose
+        working-directory: ./tests/networks/docker-compose-localnet
         run: |
-          ./gen_node_configs.sh
-          ./run_docker.sh
+          ./gen-node-configs.sh
+          ./run-docker.sh
           ./wait.sh '[[ $(curl -s -N localhost:26657/block | jq -cr '"'"'.result.block.last_commit.height'"'"') -gt 1 ]] && echo "Height is more than 1"'
 
       - name: Set up test environment
@@ -292,7 +295,7 @@ jobs:
       - name: Run test net container
         run: |
           docker run -d --rm -p "26657:26657" cheqd-testnet
-          bash tests/networks/wait_for_chain.sh
+          bash tests/networks/tools/wait-for-chain.sh
       
       - name: Set up test environment
         working-directory: ./tests/e2e
@@ -333,8 +336,8 @@ jobs:
 
       - name: Run test net container
         run: |
-          docker run -d --rm -p "26657:26657"  -p "1317:1317" cheqd-testnet
-          bash tests/networks/wait_for_chain.sh
+          docker run -d --rm -p "26657:26657" -p "1317:1317" cheqd-testnet
+          bash tests/networks/tools/wait-for-chain.sh
 
       - name: Set up and run tests
         working-directory: tests/cli

diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -20,10 +20,10 @@ jobs:
       - name: Set outputs
         id: cache
         run: |
-          # Exposes org_name/repository_name in lower case. Required by docker.
+          # Exposes org_name/repository_name in lower case. Required by Docker.
           echo "::set-output name=GITHUB_REPOSITORY_NAME::$(echo ${GITHUB_REPOSITORY,,})"
 
-          # Version is tag without `v` prefix. Reuired by deb package.
+          # Version is tag without `v` prefix. Required by Debian package.
           TAG=${GITHUB_REF/refs\/tags\//}
           echo "::set-output name=TAG::$TAG"
 
@@ -39,7 +39,7 @@ jobs:
         with:
           go-version: 1.17
 
-      - name: Get go protoc compiler plugins
+      - name: Get Go protoc compiler plugins
         env:
           GOLANG_PROTOBUF_VERSION: 1.3.5
           GOGO_PROTOBUF_VERSION: 1.3.2
@@ -78,7 +78,7 @@ jobs:
           path: build/cheqd-noded
 
   build-deb-package:
-    name: Build deb package
+    name: Build Debian package
     runs-on: ubuntu-20.04
     needs: [ build-binary, setup-workflow ]
     env:
@@ -97,36 +97,39 @@ jobs:
         run: sudo chmod +x /home/runner/go/bin/cheqd-noded
 
       - name: Make intermediate archive
-        working-directory: ./build_tools
+        working-directory: ./build-tools
         run: |
-          ./build_tar.sh ${{ env.PACKAGE_NAME }} ${{ env.VERSION }}
+          ./build-tar.sh ${{ env.PACKAGE_NAME }} ${{ env.VERSION }}
 
       - name: Install fpm
         run: |
           sudo apt-get install ruby ruby-dev rubygems build-essential
           sudo gem install --no-document fpm
 
-      - name: Build deb # Depends on tar archive
+      - name: Build Debian package # Depends on tar archive
+        working-directory: ./build-tools
         run: |
-          ./build_deb.sh ${{ env.PACKAGE_NAME }} ${{ env.VERSION }}
-        working-directory: ./build_tools
+          ./build-deb.sh ${{ env.PACKAGE_NAME }} ${{ env.VERSION }}
 
-      - name: Store deb package artifact
+      - name: Store Debian package artifact
         uses: actions/upload-artifact@v2
         with:
           name: ${{ env.PACKAGE_NAME }}_${{ env.VERSION }}_amd64.deb
-          path: build_tools/output/${{ env.PACKAGE_NAME }}_${{ env.VERSION }}_amd64.deb
+          path: build-tools/output/${{ env.PACKAGE_NAME }}_${{ env.VERSION }}_amd64.deb
 
   build-node-image:
-    name: Build cheqd-node image
+    name: Build cheqd-node Docker image
     runs-on: ubuntu-20.04
     steps:
       - uses: actions/checkout@v2
         with:
           fetch-depth: 0 # FIXME: Starport requires full repository
 
       - name: Build
-        run: docker build -f docker/cheqd_node/Dockerfile --no-cache -t cheqd-node  --build-arg UID=$(id -u) --build-arg GID=$(id -g) . # TODO: Get rid of UID and GID
+        run: |
+          export CHEQD_NODE_VERSION=local
+          docker compose --env-file docker/persistent-chains/docker-compose.env -f docker/persistent-chains/docker-compose.yml build
+          docker tag ghcr.io/cheqd/cheqd-node:local cheqd-node
 
       - name: Save
         run: docker save -o cheqd-node-image.tar cheqd-node
@@ -154,7 +157,7 @@ jobs:
         run: docker load -i cheqd-node-image.tar
 
       - name: Build
-        run: docker build -f docker/single_image_testnet/Dockerfile --no-cache -t cheqd-testnet .
+        run: docker build -f tests/networks/docker-localnet/Dockerfile --no-cache -t cheqd-testnet .
 
       - name: Save
         run: docker save -o cheqd-testnet-image.tar cheqd-testnet

diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,9 @@
+### APP-SPECIFIC EXCLUSIONS ###
+**/node_configs
+.idea
+
+### GENERAL EXCLUSIONS ###
+
 ### macOS ###
 # General
 .DS_Store
@@ -30,14 +36,6 @@ Temporary Items
 .devcontainer
 .vscode
 
-# Application specific
-vue/node_modules
-vue/src/store/generated
-vue/dist
-secret.yml
-**/node_configs
-.idea
-
 # Python
 .pytest_cache
 **/__pycache__

diff --git a/SUMMARY.md b/SUMMARY.md
@@ -21,9 +21,9 @@
   * [Using cheqd Cosmos CLI for token transactions](docs/cheqd-cli/cheqd-cli-token-transactions.md)
 * [Building from source](docs/build-and-networks/README.md)
   * [Building and testing](docs/build-and-networks/README.md)
-  * [cheqd-testsnet docker image](docs/build-and-networks/local-docker-network.md)
-  * [Docker Compose Based Localnet](docs/build-and-networks/docker-compose.md)
-  * [Setting up a new network](docs/build-and-networks/build-and-networks.md)
+  * [Docker Based Localnet](docs/build-and-networks/local-docker-network.md)
+  * [Docker Compose Based Localnet](docs/build-and-networks/local-docker-compose-network.md)
+  * [Setting up a new network](docs/build-and-networks/manual-network-setup.md)
 * [Client-app Identity APIs](docs/identity-api/README.md)
   * [Error messages](docs/identity-api/identity-api-error-messages.md)
 

diff --git a/build_tools/build_deb.sh → build-tools/build-deb.sh b/build_tools/build_deb.sh → build-tools/build-deb.sh
diff --git a/build_tools/build_tar.sh → build-tools/build-tar.sh b/build_tools/build_tar.sh → build-tools/build-tar.sh
diff --git a/build_tools/common.sh → build-tools/common.sh b/build_tools/common.sh → build-tools/common.sh
diff --git a/build_tools/postinst → build-tools/postinst b/build_tools/postinst → build-tools/postinst
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 
 CHEQD_USER_NAME=cheqd
 

diff --git a/build_tools/postremove → build-tools/postremove b/build_tools/postremove → build-tools/postremove