demo-md-walkthrough-testing

Summary:

- Support for simple testing of walkthroughs written in markdown, with annotations.
- Whole table match scenario.
- Nonempty table match scenario.
- Skip testing on files named `README.md`.
- Runnable from CI, with tag of form: `scenario-<stackql_build_run_number>-<anything>`.

.github/workflows/scenario.yaml

@@ -0,0 +1,71 @@
+name: scenario
+
+on:
+  push:
+    tags:
+      - 'scenario*'
+
+env:
+  TAG_NAME: ${{ github.ref_name }}
+
+jobs:
+  scenario-testing:
+    runs-on: ubuntu-latest
+    timeout-minutes: ${{ vars.DEFAULT_JOB_TIMEOUT_MIN == '' && 120 || vars.DEFAULT_JOB_TIMEOUT_MIN }}
+    steps:
+      - name: Ref Parse
+        run: |
+          {
+            echo "runID=$(echo -n '${{ github.ref_name }}' | cut -d '-' -f 2)"
+          } >> "${GITHUB_ENV}"
+
+      - name: Check out code into the Go module directory
+        uses: actions/[email protected]
+
+      - name: Download Artifact
+        uses: actions/[email protected]
+        with:
+          name: stackql_linux_amd64
+          path: build
+          github-token: ${{ secrets.CI_STACKQL_PACKAGE_DOWNLOAD_TOKEN }}
+          repository: stackql/stackql
+          run-id: ${{ env.runID }}
+
+      - name: Stackql permissions
+        run:  |
+          sudo chmod a+rwx build/stackql
+          ls -al build/stackql
+          {
+            echo "$(pwd)/build"
+          } >> "${GITHUB_PATH}"
+
+      - name: Check Stackql Version
+        run: |
+          stackql --version
+      
+      - name: Persist secrets
+        run: |
+          echo "$GCP_RO_SECRET" >> cicd/keys/testing/google-ro-credentials.json
+        shell: bash
+        env:
+          GCP_RO_SECRET: ${{ secrets.CI_SCENARIO_GCP_RO_SECRET }}
+
+      - name: Install Python dependencies
+        run: |
+          pip3 install -r cicd/requirements.txt
+      
+      - name: Run Walkthrough Scenarios
+        run: |
+          python3 test/python/markdown_testing/markdown_testing.py 2>&1 | tee cicd/log/markdown-testing-results.log 
+      
+      - name: Upload Test Results
+        uses: actions/[email protected]
+        with:
+          name: scenario_test_results
+          path: cicd/log/markdown-testing-results.log
+
+      - name: Cleanup
+        if: always()
+        run: |
+          rm -rf cicd/keys/testing
+

cicd/requirements.txt

@@ -1,4 +1,5 @@
 Flask==3.0.3
+mistune==3.0.2
 psycopg2-binary>=2.9.9
 psycopg[binary]>=3.1.16
 PyYaml>=6.0.1

docs/walkthroughs/README.md

@@ -0,0 +1,22 @@
+
+# `stackql` walkthroughs
+
+All markdown documents hereunder, execept those named `README.md`, 
+are **provably working** examples of `stackql` in action.
+These materials serve as useful examples and reference materials for
+using `stackql`.  If you have some use case that you would like to see added here; please let us know! 
+
+
+All walkthrough files are testable with CI, leveraging annotations (eg: code block info strings)
+in order to setup, run, verify and tear down testing scenarios.  The tests *can* be run:
+
+- Locally and manually, on your own machine.  That's the whole idea; please follow the instructions, mix and match, and let us know any ideas that occur.
+- Directly from CI.  Reports are generated and archived.
+- From test harnesses, such as robot framework.  This has not yet been implemented.
+
+## Running from CI
+
+The canonical, **ruleset-protected** tag form is `scenario-<run_number>-<anything>`.  At this stage, `run_number` must refer to a `stackql` run for which a `linux` `amd64` stackql binary archive is present at the time the tag is run.  
+
+
+

docs/walkthroughs/get-google-accelerator-types.md

@@ -0,0 +1,70 @@
+
+## Setup
+
+First, create a google service account key using the GCP Console, per [the GCP documentation](https://cloud.google.com/iam/docs/keys-create-delete).  Grant the service account at least `Viewer` role equivalent privileges, per [the GCP dumentation](https://cloud.google.com/iam/docs/create-service-agents#grant-roles).
+
+Then, do this in bash:
+
+```bash setup stackql-shell credentials-path=cicd/keys/testing/google-ro-credentials.json app-root-path=./test/tmp/.get-google-accel.stackql
+
+export GOOGLE_CREDENTIALS="$(cat <credentials-path>)";
+
+stackql shell --approot=<app-root-path>
+```
+
+## Method
+
+Do this in the `stackql` shell, replacing `<project>` with your GCP project name, and `<zone>` as desired, eg: `australia-southeast1-a`:
+
+```sql stackql-shell input required project=stackql-demo zone=australia-southeast1-a
+
+registry pull google;
+
+select 
+  name, 
+  kind 
+FROM google.compute.accelerator_types 
+WHERE 
+  project = '<project>' 
+  AND zone = '<zone>'
+ORDER BY
+  name desc
+;
+
+```
+
+## Result
+
+
+You will see something very much like this included in the output, presuming you have one VM (if you have zero, only the headers should appper, more VMs means more rows):
+
+```sql expectation stdout-contains-all
+|---------------------|-------------------------|
+|        name         |          kind           |
+|---------------------|-------------------------|
+| nvidia-tesla-t4-vws | compute#acceleratorType |
+|---------------------|-------------------------|
+| nvidia-tesla-t4     | compute#acceleratorType |
+|---------------------|-------------------------|
+| nvidia-tesla-p4-vws | compute#acceleratorType |
+|---------------------|-------------------------|
+| nvidia-tesla-p4     | compute#acceleratorType |
+|---------------------|-------------------------|
+```
+
+<!---  EXPECTATION
+google\ provider,\ version\ 'v24.11.00274'\ successfully\ installed
+goodbye
+-->
+
+<x-expectation style="display: none;">
+<stdout-contains-nonempty-table></stdout-contains-nonempty-table>
+</x-expectation>
+
+## Cleanup
+
+```bash teardown best-effort app-root-path=./test/tmp/.get-google-accel.stackql
+
+rm -rf <app-root-path>
+
+```

docs/walkthroughs/get-google-vms.md

@@ -0,0 +1,62 @@
+
+## Setup
+
+First, create a google service account key using the GCP Console, per [the GCP documentation](https://cloud.google.com/iam/docs/keys-create-delete).  Grant the service account at least `Viewer` role equivalent privileges, per [the GCP dumentation](https://cloud.google.com/iam/docs/create-service-agents#grant-roles).
+
+Then, do this in bash:
+
+```bash setup stackql-shell credentials-path=cicd/keys/testing/google-ro-credentials.json app-root-path=./test/tmp/.get-google-vms.stackql
+
+export GOOGLE_CREDENTIALS="$(cat <credentials-path>)";
+
+stackql shell --approot=<app-root-path>
+```
+
+## Method
+
+Do this in the `stackql` shell, replacing `<project>` with your GCP project name, and `<zone>` as desired, eg: `australia-southeast1-a`:
+
+```sql stackql-shell input required project=stackql-demo zone=australia-southeast1-a
+
+registry pull google;
+
+select 
+  name, 
+  id 
+FROM google.compute.instances 
+WHERE 
+  project = '<project>' 
+  AND zone = '<zone>'
+;
+
+```
+
+## Result
+
+
+You will see something very much like this included in the output, presuming you have one VM (if you have zero, only the headers should appper, more VMs means more rows):
+
+```sql stackql stdout expectation stdout-table-contains-data
+|--------------------------------------------------|---------------------|
+|                       name                       |         id          |
+|--------------------------------------------------|---------------------|
+| any-compute-cluster-1-default-abcd-00000001-0001 | 1000000000000000001 |
+|--------------------------------------------------|---------------------|
+```
+
+<!---  EXPECTATION
+google\ provider,\ version\ 'v24.11.00274'\ successfully\ installed
+goodbye
+-->
+
+<x-expectation style="display: none;">
+<stdout-contains-nonempty-table></stdout-contains-nonempty-table>
+</x-expectation>
+
+## Cleanup
+
+```bash teardown best-effort app-root-path=./test/tmp/.get-google-vms.stackql
+
+rm -rf <app-root-path>
+
+```