Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: Test Models Against the Full Docker Stack #3183

Closed
Closed
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 150 additions & 0 deletions .github/workflows/docker-stack-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
name: Docker-Stack-Test

on:
push:
branches:
- master
- develop
- Docker-Stack-Testing


tags:
- '*'

pull_request:

merge_group:

issue_comment:
types:
- created

env:
R_LIBS_USER: /usr/local/lib/R/site-library
LC_ALL: en_US.UTF-8
NCPUS: 2
PGHOST: postgres
CI: true

jobs:


# ----------------------------------------------------------------------
# DOCKER STACK TESTS
# ----------------------------------------------------------------------
Stack-Test:
if: github.event_name != 'issue_comment' || startsWith(github.event.comment.body, '/build')
allgandalf marked this conversation as resolved.
Show resolved Hide resolved
runs-on: ubuntu-latest
env:
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}

strategy:
fail-fast: false
matrix:
name:
- sipnet_r136

include:
- name: sipnet_r136
modelID: "1000000014"
siteID: "772"
siteName: "Niwot Ridge Forest/LTER NWT1 (US-NR1)"
pftName: "temperate.deciduous"
metData: "AmerifluxLBL"
startDate: "2004/01/01"
endDate: "2004/12/31"

#Add more models below in the above format





services:
postgres:
image: mdillon/postgis:9.5
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

container:
image: pecan/depends
Comment on lines +38 to +39
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm rusty on the details here -- Does running this test inside the depends container mean these deps are used directly, or do we wind up pulling the whole container again to be run inside itself? Would it work to not specify a container at all let the Docker stack run directly on the host?


steps:
# checkout source code
- name: work around https://github.com/actions/checkout/issues/766
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
- uses: actions/checkout@v3
with:
set-safe-directory: false

- name: Set up Docker Compose
run: |
sudo apt-get update
sudo apt-get -y install docker-compose
- name: Build and run containers
run: |
docker system prune --all --force
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is a prune needed here? Won't it be running in a fresh VM (presumably with ~empty docker cache) every time?

docker-compose up -d postgres
docker run --rm --network pecan_pecan pecan/db

docker-compose run bety user guestuser guestuser "Guest User" [email protected] 4 4


docker-compose run bety user carya illinois "Carya Demo User" [email protected] 1 1
docker run --rm --network pecan_pecan --volume pecan_pecan:/data --env FQDN=docker pecan/data:develop

cp docker/env.example .env
echo "COMPOSE_PROJECT_NAME=pecan" >> .env
echo "PECAN_VERSION=develop" >> .env

docker-compose up -d



- name: Wait for services to start
run: |

docker-compose ps
sleep 20 # Adjust the duration as needed
docker-compose ps
sleep 20
docker-compose ps
sleep 20
Comment on lines +78 to +83
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be a loop over an actual status check; maybe something like while [[ $(docker compose ps --format '{{.State}}'|sort|uniq) != "running" ]]; do sleep 10; done




- name: Display running containers
run: |
docker-compose ps
docker network ls
docker network inspect bridge
Comment on lines +86 to +90
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How useful is this step for unattended CI run? Will these outputs be helpful for debugging or will they just be filling the log up?




# run tests against the full docker stack
- name: Run cURL command
run: |
curl -v -L -X POST -H "Host: pecan.localhost" \
-F 'hostname=docker' \
-F 'modelid=${{ matrix.modelID }}' \
-F 'sitegroupid=1' \
-F 'siteid=${{ matrix.siteID }}' \
-F 'sitename=${{ matrix.siteName }}' \
-F 'pft[]=${{ matrix.pftName }}' \
-F 'start=${{ matrix.startDate }}' \
-F 'end=${{ matrix.endDate }}' \
-F 'input_met=${{ matrix.metData }}' \
-F 'email=' \
-F 'notes=' \
'http://172.17.0.1/pecan/04-runpecan.php'



curl -v -L -H "Host: pecan.localhost" \
'http://172.17.0.1/pecan/08-finished.php?workflowid=99000000001&hostname=docker&loglines=100'

sleep 200
curl -v -L -H "Host: pecan.localhost" \
'http://172.17.0.1/pecan/08-finished.php?workflowid=99000000001&hostname=docker&loglines=100'

curl -v -L -H "Host: pecan.localhost" \
'http://172.17.0.1/pecan/dataset.php?workflowid=99000000001&type=file&name=out/99000000001/sipnet.out'
1 change: 1 addition & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,7 @@ services:
api:
image: pecan/api:${PECAN_VERSION:-latest}
user: "${UID:-1001}:${GID:-1001}"
restart: unless-stopped
Copy link
Collaborator Author

@allgandalf allgandalf Jun 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can be useful to automatically restart the api container if it ever fails when we bring up the whole docker stack

networks:
- pecan
environment:
Expand Down