feat(docker): add docker-compose file to run CI locally (#8209)

* ref(docker): Consolidate all tests in the `entrypoint.sh` script * fix(ci): missing docker argument * fix(ci): do not blank `ENTRYPOINT_FEATURES` vairable * fix(docker): do not add `shielded-scan` as part of `TEST_FEATURES` * chore: remove misleading comment * feat(docker): add `docker-compose` file to run CI locally * fix(docs): do not remove previous sections until we have a substitute for users * fix(docs): revert some changes * fix: typos * ref(docker): Consolidate all tests in the `entrypoint.sh` script * fix(ci): missing docker argument * fix(ci): do not blank `ENTRYPOINT_FEATURES` vairable * fix(docker): do not add `shielded-scan` as part of `TEST_FEATURES` * chore: remove misleading comment * Update docker/Dockerfile Co-authored-by: Arya <[email protected]> --------- Co-authored-by: Arya <[email protected]>
ZcashFoundation · Jan 30, 2024 · 18477ec · 18477ec
1 parent 184c30e
commit 18477ec
Show file tree

Hide file tree

Showing 3 changed files with 145 additions and 5 deletions.
diff --git a/book/src/user/docker.md b/book/src/user/docker.md
@@ -2,7 +2,7 @@
 
 The easiest way to run Zebra is using [Docker](https://docs.docker.com/get-docker/).
 
-We've embraced Docker in Zebra for most of the solution lifecycle, from development environments to CI (in our pipelines), and deployment to end users.
+We've embraced Docker in Zebra for most of the solution lifecycle, from development environments to CI (in our pipelines), and deployment to end users. We recommend using `docker-compose` over plain `docker` CLI, especially for more advanced use-cases like running CI locally, as it provides a more convenient and powerful way to manage multi-container Docker applications.
 
 ## Quick usage
 
@@ -63,11 +63,45 @@ cache_dir = "/var/cache/zebrad-cache"
 endpoint_addr = "127.0.0.1:9999"
 ```
 
-### Build time arguments
+### CI/CD Local Testing
+
+To run CI tests locally, which mimics the testing done in our CI pipelines on GitHub Actions, use the `docker-compose.test.yml` file. This setup allows for a consistent testing environment both locally and in CI.
+
+#### Running Tests Locally
+
+1. **Setting Environment Variables**:
+   - Modify the `test.env` file to set the desired test configurations.
+   - For running all tests, set `RUN_ALL_TESTS=1` in `test.env`.
+
+2. **Starting the Test Environment**:
+   - Use Docker Compose to start the testing environment:
+
+     ```shell
+     docker-compose -f docker/docker-compose.test.yml up
+     ```
+
+   - This will start the Docker container and run the tests based on `test.env` settings.
+
+3. **Viewing Test Output**:
+   - The test results and logs will be displayed in the terminal.
+
+4. **Stopping the Environment**:
+   - Once testing is complete, stop the environment using:
+
+     ```shell
+     docker-compose -f docker/docker-compose.test.yml down
+     ```
+
+This approach ensures you can run the same tests locally that are run in CI, providing a robust way to validate changes before pushing to the repository.
+
+### Build and Run Time Configuration
+
+#### Build Time Arguments
 
 #### Configuration
 
 - `FEATURES`: Specifies the features to build `zebrad` with. Example: `"default-release-binaries getblocktemplate-rpcs"`
+- `TEST_FEATURES`: Specifies the features for tests. Example: `"lightwalletd-grpc-tests zebra-checkpoints"`
 
 #### Logging
 
@@ -86,9 +120,7 @@ endpoint_addr = "127.0.0.1:9999"
 
 - `SHORT_SHA`: Represents the short SHA of the commit. Example: `"a1b2c3d"`
 
-### Run time variables
-
-#### Network
+#### Run Time Variables
 
 - `NETWORK`: Specifies the network type. Example: `"Mainnet"`
 
@@ -113,6 +145,8 @@ endpoint_addr = "127.0.0.1:9999"
 - `TRACING_ENDPOINT_ADDR`: Address for tracing endpoint. Example: `"0.0.0.0"`
 - `TRACING_ENDPOINT_PORT`: Port for tracing endpoint. Example: `3000`
 
+Specific tests are defined in `docker/test.env` file and can be enabled by setting the corresponding environment variable to `1`.
+
 ## Registries
 
 The images built by the Zebra team are all publicly hosted. Old image versions meant to be used by our [CI pipeline](https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/ci-integration-tests-gcp.yml) (`zebrad-test`, `lighwalletd`) might be deleted on a scheduled basis.

diff --git a/docker/docker-compose.test.yml b/docker/docker-compose.test.yml
@@ -0,0 +1,36 @@
+version: "3.8"
+
+services:
+  zebra:
+    build:
+      context: ../
+      dockerfile: docker/Dockerfile
+      target: tests
+    restart: unless-stopped
+    deploy:
+      resources:
+        reservations:
+          cpus: "4"
+          memory: 16G
+    # Change this to the command you want to run, respecting the entrypoint.sh
+    # For example, to run the tests, use the following command:
+    # command: ["cargo", "test", "--locked", "--release", "--features", "${TEST_FEATURES}", "--package", "zebrad", "--test", "acceptance", "--", "--nocapture", "--include-ignored", "sync_large_checkpoints_"]
+    volumes:
+      - zebrad-cache:/var/cache/zebrad-cache
+      - lwd-cache:/var/cache/lwd-cache
+    ports:
+      # Zebra uses the following inbound and outbound TCP ports
+      - "8232:8232" # Opens an RPC endpoint (for wallet storing and mining)
+      - "8233:8233" # Mainnet Network (for peer connections)
+      - "18233:18233" # Testnet Network
+      # - "9999:9999" # Metrics
+      # - "3000:3000" # Tracing
+    env_file:
+      - test.env
+
+volumes:
+  zebrad-cache:
+    driver: local
+
+  lwd-cache:
+    driver: local
diff --git a/docker/test.env b/docker/test.env
@@ -0,0 +1,70 @@
+RUST_LOG=info
+# This variable forces the use of color in the logs
+ZEBRA_FORCE_USE_COLOR=1
+LOG_COLOR=true
+
+###
+# Configuration Variables
+# These variables are used to configure the zebra node
+# Check the entrypoint.sh script for more details
+###
+
+# Path and name of the config file. These two have defaults set in the Dockerfile.
+ZEBRA_CONF_PATH=/etc/zebrad/zebrad.toml
+# [network]
+NETWORK=Mainnet
+ZEBRA_LISTEN_ADDR=0.0.0.0
+# [consensus]
+ZEBRA_CHECKPOINT_SYNC=true
+# [state]
+# Set this to change the default cached state directory
+ZEBRA_CACHED_STATE_DIR=/var/cache/zebrad-cache
+LIGHTWALLETD_DATA_DIR=/var/cache/lwd-cache
+# [metrics]
+METRICS_ENDPOINT_ADDR=0.0.0.0
+METRICS_ENDPOINT_PORT=9999
+# [tracing]
+LOG_COLOR=false
+TRACING_ENDPOINT_ADDR=0.0.0.0
+TRACING_ENDPOINT_PORT=3000
+# [rpc]
+RPC_LISTEN_ADDR=0.0.0.0
+# if ${RPC_PORT} is not set, it will use the default value for the current network
+# RPC_PORT=
+
+####
+# Test Variables
+# These variables are used to run tests in the Dockerfile
+# Check the entrypoint.sh script for more details
+####
+
+# Unit tests
+# TODO: These variables are evaluated to any value, even setting a NULL value will evaluate to true
+# TEST_FAKE_ACTIVATION_HEIGHTS=
+# ZEBRA_SKIP_NETWORK_TESTS
+# ZEBRA_SKIP_IPV6_TESTS
+RUN_ALL_TESTS=
+RUN_ALL_EXPERIMENTAL_TESTS=
+TEST_ZEBRA_EMPTY_SYNC=
+ZEBRA_TEST_LIGHTWALLETD=
+# Integration Tests
+# Most of these tests require a cached state directory to save the network state
+TEST_DISK_REBUILD=
+# These tests needs a Zebra cached state
+TEST_CHECKPOINT_SYNC=
+GENERATE_CHECKPOINTS_MAINNET=
+GENERATE_CHECKPOINTS_TESTNET=
+TEST_UPDATE_SYNC=
+# These tests need a Lightwalletd binary + a Zebra cached state
+TEST_LWD_RPC_CALL=
+TEST_GET_BLOCK_TEMPLATE=
+TEST_SUBMIT_BLOCK=
+# These tests need a Lightwalletd binary + Lightwalletd cached state + a Zebra cached state
+TEST_LWD_UPDATE_SYNC=
+TEST_LWD_GRPC=
+TEST_LWD_TRANSACTIONS=
+# Full sync tests
+# These tests could take a long time to run, depending on the network
+FULL_SYNC_MAINNET_TIMEOUT_MINUTES=
+FULL_SYNC_TESTNET_TIMEOUT_MINUTES=
+TEST_LWD_FULL_SYNC=