Development environment (#972)

* * Image name changed from `pocs` -> `panoptes-pocs`

* Better install options for extra features.

* * Developer option Docker moving from panoptes-utils to pocs.

* * Remove TODO about docker install. That should be handled by the script.

* * Update test files.

* * Cleaning up docker images:
  * `latest` installs the `panoptes-pocs` module from pip
  * `develop` installs via `pip install -e[google.testing]` and is used for running the CI tests.
  `developer-env` installs locally but with all options, i.e. `pip install -e[google,testing,plotting,developer]`. Also builds jupyterlab and other developer tools. Starts a jupyterlab instance by default.

* * Changelog updates.

* * `develop` image installs the module as the root user.

* * Docker build ignore git directory unless buliding locally (needed by pyscaffold and we are buliding a local develop image anyway).
* Use new `arduino-cli` installer.
* Add `bin/panoptes-develop` and `bin/wait-for-it.sh` to installed scripts.
* Adding a convenience script for building local images.

* Docker image updates

* * Pass git folder to build context for testing images.

* * $PANUSER owns $PANDIR.

* * Ignore log files..

* * Fix log files on GHA. I don't think this should be needed.

* * PANUSER owns PANDIR in develop.
* Adding the SOLVE_FIELD env var.

* * Give astrometry data all readable.

* * Docker docs

* * Updated install script.

* * Update changelog

.dockerignore

@@ -1,16 +1,18 @@
-!.git
 docs/*
-.eggs
 .idea
 .venv
 venv
-*.egg-info
+
+.git
 .github
 
 *.md
 !README*.md
 
-*.log
-*.pdf
+logs/
+**/.eggs
+**/*.pdf
+**/*.log
+**/*.egg-info
 **/*.pyc
 **/__pycache__

.github/workflows/pythontest.yaml

@@ -29,22 +29,22 @@ jobs:
     steps:
       - name: Checkout code
         uses: actions/checkout@v2
-      - name: Fetch all history for all tags and branches
-        run: git fetch --prune --unshallow
-      - name: Build pocs image
+      - name: Build panoptes-pocs image
         run: |
-          docker build -t pocs:testing -f docker/latest.Dockerfile .
-      - name: Test with pytest in pocs container
+          # Make sure git goes to the build context.
+          sed -i s'/^\.git$/\!\.git/' .dockerignore
+          docker build -t panoptes-pocs:develop -f docker/develop.Dockerfile .
+      - name: Test with pytest in panoptes-pocs container
         run: |
           mkdir -p coverage_dir && chmod 777 coverage_dir
           ci_env=`bash <(curl -s https://codecov.io/env)`
           docker run -i \
             $ci_env \
             -e REPORT_FILE="/tmp/coverage/coverage.xml" \
             --network "host" \
-            -v $PWD:/var/panoptes/logs \
+            -v $PWD/coverage_dir:/var/panoptes/logs \
             -v $PWD/coverage_dir:/tmp/coverage \
-            pocs:testing \
+            panoptes-pocs:develop \
             scripts/testing/run-tests.sh
       - name: Upload coverage report to codecov.io
         uses: codecov/codecov-action@v1
@@ -58,4 +58,4 @@ jobs:
         if: always()
         with:
           name: log-files
-          path: panoptes-testing.log
+          path: coverage_dir/panoptes-testing.log

.gitignore

@@ -1,14 +1,14 @@
 # Temporary and binary files
 *~
-*.py[cod]
 *.so
 *.cfg
 !.isort.cfg
 !setup.cfg
 *.orig
-*.log
 *.pot
-__pycache__/*
+**/*.py[cod]
+**/*.log
+**/__pycache__/*
 .cache/*
 .*.swp
 */.ipynb_checkpoints/*

.travis.yml

@@ -1,18 +1,19 @@
 dist: xenial
 sudo: required
 language: python
+addons:
+  apt:
+    packages:
+      - docker-ce
 python:
   - "3.6"
 services:
 - docker
 before_install:
-- docker pull gcr.io/panoptes-exp/pocs:latest
-- ci_env=`bash <(curl -s https://codecov.io/env)`
+- sed -i s'/^\.git$/\!\.git/' .dockerignore
+- docker build -t panoptes-pocs:develop -f ${TRAVIS_BUILD_DIR}/docker/develop.Dockerfile ${TRAVIS_BUILD_DIR}
 install: true
 script:
 - docker run -it
-  $ci_env
-  -e LOCAL_USER_ID=0
-  -v $TRAVIS_BUILD_DIR:/var/panoptes/POCS
-  gcr.io/panoptes-exp/pocs:latest
+  panoptes-pocs:develop
   scripts/testing/run-tests.sh

CHANGELOG.rst

@@ -6,17 +6,31 @@ All notable changes to this project will be documented in this file.
 The format is based on `Keep a Changelog <https://keepachangelog.com/en/1.0.0/>`__, and this project
 adheres to `Semantic Versioning <https://semver.org/spec/v2.0.0.html>`__.
 
+[0.7.5dev]
+----------
+
+Changed
+~~~~~~~
+
+* Docker image updates (#972)
+    * Updated `install-pocs.sh` script.
+    * ``latest`` installs the ``panoptes-pocs`` module from pip
+    * ``develop`` installs via ``pip install -e[google.testing]`` and is used for running the CI tests.
+    * ``developer-env`` installs locally but with all options, i.e. ``pip install -e[google,testing,plotting,developer]``. Also builds ``jupyterlab`` and other developer tools. Starts a ``jupyterlab`` instance by default.
+    * Use new ``arduino-cli`` installer.
+    * Add ``bin/panoptes-develop`` and ``bin/wait-for-it.sh`` to installed scripts.
+    * Add ``docker/setup-local-environment.sh``, a convenience script for building local images.
 
 [0.7.4] - 2020-05-31
 ----------
 
-Note that we skipped `0.7.2` and `0.7.3`.
+Note that we skipped ``0.7.2`` and ``0.7.3``.
 
 
 Bug fixes
 ~~~~~~~~~
 
-* Package name is `panoptes-pocs` for namespace consistency. (#971)
+* Package name is ``panoptes-pocs`` for namespace consistency. (#971)
 * README changed to rst. (#971)
 
 

bin/panoptes-develop

@@ -0,0 +1,38 @@
+#!/usr/bin/bash
+set -e
+
+PARAMS="$*"
+
+export PANDIR=${PANDIR:-/var/panoptes}
+export IMAGE="${IMAGE:-panoptes-pocs}"
+export TAG="${TAG:-developer-env}"
+export CONTAINER_NAME="${CONTAINER_NAME:-pocs-developer-env}"
+
+cd "${PANDIR}"
+
+CMD="docker-compose \
+    --project-directory ${PANDIR} \
+    --env-file ${PANDIR}/env \
+    -f POCS/docker/docker-compose-developer-env.yaml \
+    -p panoptes"
+
+## Add the deamon option by default.
+if [[ "$PARAMS" == "up" ]]; then
+    PARAMS="up -d"
+fi
+
+# We use a docker container for docker-compose, so we need to pass the env vars to
+# that container so it can properly place them in the docker-compose file.
+export DOCKER_RUN_OPTIONS="${DOCKER_RUN_OPTIONS:--e IMAGE=${IMAGE} -e TAG=${TAG} -e CONTAINER_NAME=${CONTAINER_NAME}}"
+
+# Run the docker-compose command with user params.
+eval "DOCKER_RUN_OPTIONS=\"${DOCKER_RUN_OPTIONS}\" ${CMD} ${PARAMS}"
+
+# If we just started the environment, try to open the browser for the user.
+if [[ "$PARAMS" == "up -d" ]]; then
+    # Prompt for password
+    "${PANDIR}/POCS/bin/wait-for-it.sh" \
+        localhost:8888 \
+        -- \
+        docker exec -it -u panoptes "${CONTAINER_NAME}" jupyter notebook list | grep http | cut -d ' ' -f 1 | xargs xdg-open
+fi

bin/wait-for-it.sh

@@ -0,0 +1,179 @@
+#!/usr/bin/env bash
+# https://github.com/vishnubob/wait-for-it
+#   Use this script to test if a given TCP host/port are available
+
+WAITFORIT_cmdname=${0##*/}
+
+echoerr() { if [[ $WAITFORIT_QUIET -ne 1 ]]; then echo "$@" 1>&2; fi }
+
+usage()
+{
+    cat << USAGE >&2
+Usage:
+    $WAITFORIT_cmdname host:port [-s] [-t timeout] [-- command args]
+    -h HOST | --host=HOST       Host or IP under test
+    -p PORT | --port=PORT       TCP port under test
+                                Alternatively, you specify the host and port as host:port
+    -s | --strict               Only execute subcommand if the test succeeds
+    -q | --quiet                Don't output any status messages
+    -t TIMEOUT | --timeout=TIMEOUT
+                                Timeout in seconds, zero for no timeout
+    -- COMMAND ARGS             Execute command with args after the test finishes
+USAGE
+    exit 1
+}
+
+wait_for()
+{
+    if [[ $WAITFORIT_TIMEOUT -gt 0 ]]; then
+        echoerr "$WAITFORIT_cmdname: waiting $WAITFORIT_TIMEOUT seconds for $WAITFORIT_HOST:$WAITFORIT_PORT"
+    else
+        echoerr "$WAITFORIT_cmdname: waiting for $WAITFORIT_HOST:$WAITFORIT_PORT without a timeout"
+    fi
+    WAITFORIT_start_ts=$(date +%s)
+    while :
+    do
+        if [[ $WAITFORIT_ISBUSY -eq 1 ]]; then
+            nc -z $WAITFORIT_HOST $WAITFORIT_PORT
+            WAITFORIT_result=$?
+        else
+            (echo > /dev/tcp/$WAITFORIT_HOST/$WAITFORIT_PORT) >/dev/null 2>&1
+            WAITFORIT_result=$?
+        fi
+        if [[ $WAITFORIT_result -eq 0 ]]; then
+            WAITFORIT_end_ts=$(date +%s)
+            echoerr "$WAITFORIT_cmdname: $WAITFORIT_HOST:$WAITFORIT_PORT is available after $((WAITFORIT_end_ts - WAITFORIT_start_ts)) seconds"
+            break
+        fi
+        sleep 1
+    done
+    return $WAITFORIT_result
+}
+
+wait_for_wrapper()
+{
+    # In order to support SIGINT during timeout: http://unix.stackexchange.com/a/57692
+    if [[ $WAITFORIT_QUIET -eq 1 ]]; then
+        timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --quiet --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT &
+    else
+        timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT &
+    fi
+    WAITFORIT_PID=$!
+    trap "kill -INT -$WAITFORIT_PID" INT
+    wait $WAITFORIT_PID
+    WAITFORIT_RESULT=$?
+    if [[ $WAITFORIT_RESULT -ne 0 ]]; then
+        echoerr "$WAITFORIT_cmdname: timeout occurred after waiting $WAITFORIT_TIMEOUT seconds for $WAITFORIT_HOST:$WAITFORIT_PORT"
+    fi
+    return $WAITFORIT_RESULT
+}
+
+# process arguments
+while [[ $# -gt 0 ]]
+do
+    case "$1" in
+        *:* )
+        WAITFORIT_hostport=(${1//:/ })
+        WAITFORIT_HOST=${WAITFORIT_hostport[0]}
+        WAITFORIT_PORT=${WAITFORIT_hostport[1]}
+        shift 1
+        ;;
+        --child)
+        WAITFORIT_CHILD=1
+        shift 1
+        ;;
+        -q | --quiet)
+        WAITFORIT_QUIET=1
+        shift 1
+        ;;
+        -s | --strict)
+        WAITFORIT_STRICT=1
+        shift 1
+        ;;
+        -h)
+        WAITFORIT_HOST="$2"
+        if [[ $WAITFORIT_HOST == "" ]]; then break; fi
+        shift 2
+        ;;
+        --host=*)
+        WAITFORIT_HOST="${1#*=}"
+        shift 1
+        ;;
+        -p)
+        WAITFORIT_PORT="$2"
+        if [[ $WAITFORIT_PORT == "" ]]; then break; fi
+        shift 2
+        ;;
+        --port=*)
+        WAITFORIT_PORT="${1#*=}"
+        shift 1
+        ;;
+        -t)
+        WAITFORIT_TIMEOUT="$2"
+        if [[ $WAITFORIT_TIMEOUT == "" ]]; then break; fi
+        shift 2
+        ;;
+        --timeout=*)
+        WAITFORIT_TIMEOUT="${1#*=}"
+        shift 1
+        ;;
+        --)
+        shift
+        WAITFORIT_CLI=("$@")
+        break
+        ;;
+        --help)
+        usage
+        ;;
+        *)
+        echoerr "Unknown argument: $1"
+        usage
+        ;;
+    esac
+done
+
+if [[ "$WAITFORIT_HOST" == "" || "$WAITFORIT_PORT" == "" ]]; then
+    echoerr "Error: you need to provide a host and port to test."
+    usage
+fi
+
+WAITFORIT_TIMEOUT=${WAITFORIT_TIMEOUT:-15}
+WAITFORIT_STRICT=${WAITFORIT_STRICT:-0}
+WAITFORIT_CHILD=${WAITFORIT_CHILD:-0}
+WAITFORIT_QUIET=${WAITFORIT_QUIET:-0}
+
+# check to see if timeout is from busybox?
+WAITFORIT_TIMEOUT_PATH=$(type -p timeout)
+WAITFORIT_TIMEOUT_PATH=$(realpath $WAITFORIT_TIMEOUT_PATH 2>/dev/null || readlink -f $WAITFORIT_TIMEOUT_PATH)
+if [[ $WAITFORIT_TIMEOUT_PATH =~ "busybox" ]]; then
+        WAITFORIT_ISBUSY=1
+        WAITFORIT_BUSYTIMEFLAG="-t"
+
+else
+        WAITFORIT_ISBUSY=0
+        WAITFORIT_BUSYTIMEFLAG=""
+fi
+
+if [[ $WAITFORIT_CHILD -gt 0 ]]; then
+    wait_for
+    WAITFORIT_RESULT=$?
+    exit $WAITFORIT_RESULT
+else
+    if [[ $WAITFORIT_TIMEOUT -gt 0 ]]; then
+        wait_for_wrapper
+        WAITFORIT_RESULT=$?
+    else
+        wait_for
+        WAITFORIT_RESULT=$?
+    fi
+fi
+
+if [[ $WAITFORIT_CLI != "" ]]; then
+    if [[ $WAITFORIT_RESULT -ne 0 && $WAITFORIT_STRICT -eq 1 ]]; then
+        echoerr "$WAITFORIT_cmdname: strict mode, refusing to execute subprocess"
+        exit $WAITFORIT_RESULT
+    fi
+    exec "${WAITFORIT_CLI[@]}"
+else
+    exit $WAITFORIT_RESULT
+fi