Add Metabase tests (#20)

--------- Co-authored-by: Joaquin Colacci <[email protected]> Co-authored-by: metamben <[email protected]>
MaterializeInc · Aug 21, 2023 · 54c7e18 · 54c7e18
1 parent 3493e59
commit 54c7e18
Show file tree

Hide file tree

Showing 16 changed files with 577 additions and 245 deletions.
diff --git a/.github/deps.edn b/.github/deps.edn
@@ -0,0 +1,4 @@
+{:aliases
+ {:user/materialize
+  {:extra-paths ["PWD/modules/drivers/materialize/test"]
+   :extra-deps  {metabase/materialize {:local/root "PWD/modules/drivers/materialize"}}}}}
diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
@@ -0,0 +1,89 @@
+name: Tests
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - master
+    paths-ignore:
+      - "**.md"
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Metabase Repo
+        uses: actions/checkout@v3
+        with:
+          repository: metabase/metabase
+          ref: v0.46.7
+
+      - name: Checkout Driver Repo
+        uses: actions/checkout@v3
+        with:
+          path: modules/drivers/materialize
+
+      - name: Set up JDK 11
+        uses: actions/setup-java@v2
+        with:
+          distribution: temurin
+          java-version: 11
+
+      - name: Add Materialize TLS instance to /etc/hosts
+        run: |
+          sudo echo "127.0.0.1 materialize" | sudo tee -a /etc/hosts
+
+      - name: Start Materialize in Docker
+        uses: isbang/[email protected]
+        with:
+          compose-file: "modules/drivers/materialize/docker-compose.yml"
+          services: |
+            materialize
+            init
+
+      # Apply the scripts/exclude_tests.diff patch to exclude tests that are not relevant to Materialize
+      - name: Apply exclude_tests.diff
+        run: |
+          git apply modules/drivers/materialize/scripts/exclude_tests.diff
+
+      - name: Install Clojure CLI
+        run: |
+          curl -O https://download.clojure.org/install/linux-install-1.11.1.1262.sh &&
+          sudo bash ./linux-install-1.11.1.1262.sh
+
+      - name: Setup Node
+        uses: actions/setup-node@v2
+        with:
+          node-version: "16"
+          cache: "yarn"
+
+      - name: Get M2 cache
+        uses: actions/cache@v2
+        with:
+          path: |
+            ~/.m2
+            ~/.gitlibs
+          key: ${{ runner.os }}-materialize-${{ hashFiles('**/deps.edn') }}
+
+      - name: Prepare stuff for pulses
+        run: yarn build-static-viz
+
+      # Use custom deps.edn containing "user/materialize" alias to include driver sources
+      - name: Run tests
+        run: |
+          mkdir -p /home/runner/.config/clojure
+          cat modules/drivers/materialize/.github/deps.edn | sed -e "s|PWD|$PWD|g" > /home/runner/.config/clojure/deps.edn
+          DRIVERS=materialize clojure -X:dev:drivers:drivers-dev:test:user/materialize
+
+      - name: Build Materialize driver
+        run: |
+          echo "{:deps {metabase/materialize {:local/root \"materialize\" }}}" > modules/drivers/deps.edn
+          bin/build-driver.sh materialize
+          ls -lah resources/modules
+
+      - name: Archive driver JAR
+        uses: actions/upload-artifact@v3
+        with:
+          name: materialize.metabase-driver.jar
+          path: resources/modules/materialize.metabase-driver.jar
diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,5 @@
 /target
 .clj-kondo/
 .lsp/
+.cpcache/
+.build
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -0,0 +1,114 @@
+## Getting started
+
+* Please report any issues you encounter during operations.
+* Feel free to create a pull request, preferably with a test or five.
+
+## Setting up a development environment
+
+### Requirements
+
+* Clojure 1.11+
+* OpenJDK 17
+* Node.js 16.x
+* Yarn
+
+For testing: Docker Compose
+
+Please refer to the extensive documentation available on the Metabase website: [Guide to writing a Metabase driver](https://www.metabase.com/docs/latest/developers-guide/drivers/start.html)
+
+Materialize driver's code should be inside the main Metabase repository checkout in `modules/drivers/materialize` directory.
+
+The easiest way to set up a development environment is as follows (mostly the same as in the [CI](https://github.com/MaterializeInc/metabase-materialize-driver/blob/master/.github/workflows/tests.yml)):
+
+* Clone Metabase and Materialize driver repositories
+```bash
+git clone https://github.com/metabase/metabase.git
+cd metabase
+checkout v0.46.7
+git clone https://github.com/MaterializeInc/metabase-materialize-driver.git modules/drivers/materialize
+```
+
+* Create custom Clojure profiles, you can get it using the following command:
+
+```bash
+cat modules/drivers/materialize/.github/deps.edn | sed -e "s|PWD|$PWD|g" | tr -d '\n'
+```
+
+Modifying `~/.clojure/deps.edn` will create two useful profiles: `user/materialize` that adds driver's sources to the classpath, and `user/test` that includes all the Metabase tests that are guaranteed to work with the driver.
+
+* Install the Metabase dependencies:
+
+```bash
+clojure -X:deps:drivers prep
+```
+
+* Build the frontend:
+
+```bash
+yarn && yarn build-static-viz
+```
+
+* Add `/etc/hosts` entry
+
+Required for TLS tests.
+
+```bash
+sudo -- sh -c "echo 127.0.0.1 materialize >> /etc/hosts"
+```
+
+* Start Materialize as a Docker container
+
+```bash
+docker compose -f modules/drivers/materialize/docker-compose.yml up -d materialize init
+```
+
+Now, you should be able to run the tests:
+
+```bash
+mz_deps=$(cat modules/drivers/materialize/.github/deps.edn | sed -e "s|PWD|$PWD|g" | tr -d '\n')
+DRIVERS=materialize clojure -Sdeps ${mz_deps} -X:dev:drivers:drivers-dev:test:user/materialize
+```
+
+you can see that we have our profiles `:user/materialize:user/test` added to the command above, and with `DRIVERS=materialize` we instruct Metabase to run the tests only for Materialize.
+
+> **Note** Omitting `DRIVERS` will run the tests for all the built-in database drivers.
+
+If you want to run tests for only a specific test:
+
+```bash
+mz_deps=$(cat modules/drivers/materialize/.github/deps.edn | sed -e "s|PWD|$PWD|g" | tr -d '\n')
+DRIVERS=materialize clojure -Sdeps ${mz_deps} -X:dev:drivers:drivers-dev:test:user/materialize :only metabase.query-processor.middleware.parameters.mbql-test
+```
+
+## Building a jar
+
+You need to add an entry for Materialize in `modules/drivers/deps.edn`
+
+```clj
+{:deps
+ {...
+  metabase/materialize {:local/root "materialize"}
+  ...}}
+```
+
+or just run this from the root Metabase directory, overwriting the entire file:
+
+```bash
+echo "{:deps {metabase/materialize {:local/root \"materialize\" }}}" > modules/drivers/deps.edn
+```
+
+Now, you should be able to build the final jar:
+
+```bash
+bin/build-driver.sh materialize
+```
+
+As the result, `resources/modules/materialize.metabase-driver.jar` should be created.
+
+For smoke testing, there is a Metabase with the link to the driver available as a Docker container:
+
+```bash
+docker compose -f modules/drivers/materialize/docker-compose.yml up -d metabase
+```
+
+It should pick up the driver jar as a volume.
diff --git a/Dockerfile b/Dockerfile
@@ -1,5 +1,5 @@
 ARG METABASE_VERSION=latest
 FROM metabase/metabase:${METABASE_VERSION}
 
-ADD target/dist/materialize-driver.jar /plugins/
+ADD .build/materialize-driver.jar /plugins/
 RUN chmod 744 /plugins/materialize-driver.jar
diff --git a/README.md b/README.md
@@ -1,4 +1,4 @@
-# metabase-materialize-driver
+# Materialize driver for Metabase
 
 The `metabase-materialize-driver` lets
 [Metabase](https://github.com/metabase/metabase) connect to an instance of
@@ -26,85 +26,18 @@ connect:
 | Port              | 6875      |
 | Database username | default   |
 | Database password | default   |
+| Cluster name      | default   |
 
 [releases]: https://github.com/MaterializeInc/metabase-materialize-driver/releases
 [materialize/metabase]: https://hub.docker.com/repository/docker/materialize/metabase
 
+## Choosing the Right Version
 
-### With Docker
+Metabase Release | Driver Version
+---------------- | --------------
+v0.46.7          | v0.1.0
 
-To generate a Metabase Docker image containing the Materialize driver, use [our
-Dockerfile](Dockerfile) and run the following command from your terminal:
+## Contributing
 
-```shell script
-cd /path/to/metabase-materialize-driver
-bin/build.sh --build
-```
-
-Then, to start the container, run:
-```shell script
-docker run --rm -p 3000:3000 --name metabase materialize/metabase:latest
-```
-
-Once it's finished loading, you can access Metabase at <localhost:3000>.
-
-**NB: If you are running this on a Mac and running Materialize locally, you can
-connect by substituting the following connection information:**
-
-| Field   | Value                |
-| ------- |:--------------------:|
-| Host    | [host.docker.internal](https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach/24326540#24326540) |
-
-
-### Without Docker
-
-Use this driver in two steps:
-
-1. Download a copy of the driver by running:
-   ```shell script
-   bin/build.sh --release v0.1.0 --no-docker
-   ```
-   or visiting the [releases](./releases) page and downloading one of the
-   jars
-
-1. Move the downloaded copy of the plugin (from the previous step) into the
-   `/plugins` directory of your Metabase instance.
-
-For more info, check out these resources:
-* [Managing databases in Metabase](https://www.metabase.com/docs/latest/administration-guide/01-managing-databases.html)
-* [Driver plugins in Metabase](https://github.com/metabase/metabase/wiki/Writing-a-Driver:-Packaging-a-Driver-&-Metabase-Plugin-Basics)
-
-
-## To Build the Driver
-
-**NB: These steps are not necessary to use the `metabase-materialize-driver`!
-They are only for the curious or those attempting to make updates. To just use
-the driver, stick to `How to use the driver` above.**
-
-### Step 1: Build and move the metabase-materialize-driver
-
-1. Once we've successfully completed copying over the code from the forked
-   Driver, we're ready to build the `metabase-materialize-driver`.
-
-    ```shell script
-    cd /path/to/metabase-materialize-driver
-    lein clean && lein uberjar
-    ```
-    When the `uberjar` command finishes executing, it will output the path to
-    the newly created uberjar like:
-
-    ```shell script
-    Compiling metabase.driver.materialize
-    Created /path/to/metabase-materialize-driver/target/materialize-driver-0.1.0-SNAPSHOT.jar
-    Created /path/to/metabase-materialize-driver/target/    materialize-driver-0.1.0-SNAPSHOT-standalone.jar
-    ```
-
-1. Move the `standalone.jar` (the uberjar) over to the `/plugin` directory of
-   our local copy of Metabase:
-
-    ```shell script
-    cp /path/to/metabase-materialize-driver/target/ materialize-driver-0.1.0-SNAPSHOT-standalone.jar /path/to/metabase/plugins/
-    ```
-
-Once the `metabase-materialize-driver` jar is in the `/plugins` directory,
-Metabase will register the plugin on startup!
+Check out our [contributing guide](CONTRIBUTING.md) for more information on how
+to contribute to this project.