Add scripts for starting POCS automatically - WIP (#475)

These scripts are designed for running:

* ZeroMQ message forwarders
* PAWS Web Server
* PEAS Shell
* POCS Shell
* Viewing the logs of the two shells

They are each run in a separate terminal within
tmux, a terminal multiplexer.

These were originally developed for PAN008, and this PR
is for making them more generally applicable.

This is a WIP because I've not yet installed them on
PAN008 and tested them.

scripts/startup/README.md

@@ -0,0 +1,47 @@
+These scripts support starting the PANOPTES software automatically
+when the Linux computer that runs a PANOPTES starts or restarts.
+
+First we need to have the PANOPTES environement variables set in all
+the shells that are involved in running PANOPTES. The simplest
+solution is to set them globally on the system. For example,
+on Ubuntu:
+
+```bash
+sudo cp $POCS/scripts/startup/set_panoptes_env_vars.sh \
+        /etc/profile.d/
+sudo chmod +x /etc/profile.d/set_panoptes_env_vars.sh
+```
+
+This will then be executed when any user logs in. The exact means
+of doing this varies on different versions of Unix/Linux.
+
+Next, we need to arrange for `su_panoptes.sh` to be executed when
+the operating system launches. First copy it to /etc:
+
+```bash
+sudo cp $POCS/scripts/startup/su_panoptes.sh \
+        /etc/
+sudo chmod +x /etc/su_panoptes.sh
+```
+
+Next, edit `/etc/rc.local`, adding a command to run `su_panoptes.sh`.
+For example, adding it to the default Ubuntu `rc.local`:
+
+```bash
+#!/bin/sh -e
+#
+# rc.local
+#
+# This script is executed at the end of each multiuser runlevel.
+# Make sure that the script will "exit 0" on success or any other
+# value on error.
+#
+# In order to enable or disable this script just change the execution
+# bits.
+#
+# By default this script does nothing.
+
+/etc/su_panoptes.sh >> /tmp/su_panoptes.log 2>&1
+
+exit 0
+```

scripts/startup/set_panoptes_env_vars.sh

@@ -0,0 +1,11 @@
+#!/bin/sh
+
+# This file is to be copied into /etc/profile.d/ and SOURCED by
+# /etc/rc.local. We have the shebang line at the top to help shellcheck
+# recognize the type of script.
+
+export PANUSER=panoptes          # User that runs PANOPTES software.
+export PANDIR=/var/panoptes      # Main directory.
+export PANLOG="${PANDIR}/logs"   # Log file storage.
+export POCS="${PANDIR}/POCS"     # PANOPTES Observatory Control Software
+export PAWS="${PANDIR}/PAWS"     # PANOPTES Web Interface

scripts/startup/start_log_viewer.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+WINDOW="${1}"
+LOGFILE="${2}"
+echo "Running $(basename "${0}") at $(date), WINDOW=${WINDOW}, LOGFILE=${LOGFILE}"
+
+# Wait for bash to be ready (not necessary, but makes
+# the window look tidier when you attach later).
+sleep 1s
+
+tmux send-keys -t "${WINDOW}" "date" C-m
+tmux send-keys -t "${WINDOW}" "cd \"${PANLOG}\"" C-m
+tmux send-keys -t "${WINDOW}" "less --follow-name \"${LOGFILE}\"" C-m
+sleep 2s
+tmux send-keys -t "${WINDOW}" "F"
+
+echo "Done at $(date)"

scripts/startup/start_messaging_hub.sh

@@ -0,0 +1,14 @@
+#!/bin/bash -ex
+
+WINDOW="${1}"
+echo "Running $(basename "${0}") at $(date), WINDOW=${WINDOW}"
+
+# Wait for bash to be ready (not necessary, but makes
+# the window look tidier when you attach later).
+sleep 1s
+
+tmux send-keys -t "${WINDOW}" "date" C-m
+tmux send-keys -t "${WINDOW}" \
+     "python $POCS/scripts/run_messaging_hub.py --from_config" C-m
+
+echo "Done at $(date)"

scripts/startup/start_panoptes_in_tmux.sh

@@ -0,0 +1,76 @@
+#!/bin/bash -e
+
+# This script is designed to be run inside tmux, launched by
+# tmux_launch.sh. Make sure that is so.
+echo "Running ${BASH_SOURCE[0]} at $(date)"
+if [[ -z "${STARTUP_LOG_DIR_SLASH}" ]] ; then
+  echo "This script must be run by tmux_launch.sh"
+  exit 1
+fi
+
+NOW="$(date +%Y%m%d-%H%M%S-%Z)"
+LOG_FILE="${STARTUP_LOG_DIR_SLASH}$(basename "${BASH_SOURCE[0]}" .sh)-${NOW}.log"
+
+exec 2> "${LOG_FILE}"  # send stderr to a log file
+exec 1>&2              # send stdout to the same log file
+set -x
+
+echo "Running ${BASH_SOURCE[0]} at $(date)"
+
+# Record important debugging info into the log file. These
+# environment variables do NOT all have to be set. Note that
+# this script is executed inside of a tmux session, which
+# SHOULD be treated as interactive (i.e. have the full .bashrc
+# executed). Let's check (PATH and conda info are the giveaway).
+
+echo "Current dir: $(pwd)"
+echo "Current user: ${USER}"
+echo "Current path: ${PATH}"
+echo "PANUSER: ${PANUSER}"
+echo "PANDIR: ${PANDIR}"
+echo "PANLOG: ${PANLOG}"
+echo "POCS: ${POCS}"
+echo "PAWS: ${PAWS}"
+echo "PIAA: ${PIAA}"
+echo "MATPLOTLIBRC: ${MATPLOTLIBRC}"
+# $- expands to the current option flags as specified upon invocation, by the
+# set built-in command, or those set by the shell itself (such as the -i).
+echo '$-:' "$-"
+# Just in case conda isn't setup as expected, don't die here.
+(set +e ; conda info)
+
+# We get noisy complaints from astroplan about the IERS Bulletin A
+# being too old, and this can cause some concern for those running
+# this software. Avoid these by downloading fresh data,; ignore
+# errors, such as due to the lack of an internet connection.
+(set +e ; python "${POCS}/pocs/utils/data.py")
+
+# Create a window running the zeromq message forwarders.
+# These provide connections between message publishers and subscribers.
+tmux new-window -n messaging
+./start_messaging_hub.sh :messaging
+
+# Start PAWS, the PANOPTES Administrative Web Server.
+tmux new-window -n paws
+./start_paws.sh :paws
+
+# Start PEAS, the PANOPTES Environmental Analysis System
+# (primarily takes care of reading from the sensors and loading
+# the data into a Mongo Db).
+tmux new-window -n peas
+./start_peas.sh :peas
+
+# Start POCS, the PANOPTES Observatory Control System,
+# the main software we're interested in having running.
+tmux new-window -n pocs
+./start_pocs.sh :pocs
+
+# Monitor the POCS log file.
+tmux new-window -n log_pocs
+./start_log_viewer.sh :log_pocs "${PANLOG}/pocs_shell-all.log"
+
+# Monitor the PEAS log file.
+tmux new-window -n log_peas
+./start_log_viewer.sh :log_peas "${PANLOG}/peas_shell-all.log"
+
+exit

scripts/startup/start_paws.sh

@@ -0,0 +1,13 @@
+#!/bin/bash -e
+
+WINDOW="${1}"
+echo "Running $(basename "${0}") at $(date), WINDOW=${WINDOW}"
+
+# Wait for bash to be ready (not necessary, but makes
+# the window look tidier when you attach later).
+sleep 1s
+
+tmux send-keys -t "${WINDOW}" "date" C-m
+tmux send-keys -t "${WINDOW}" "python $PAWS/app.py" C-m
+
+echo "Done at $(date)"

scripts/startup/start_peas.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+WINDOW="${1}"
+echo "Running $(basename "${0}") at $(date), WINDOW=${WINDOW}"
+
+# Wait for bash to be ready (not necessary, but makes
+# the window look tidier when you attach later).
+sleep 1s
+
+tmux send-keys -t "${WINDOW}" "date" C-m
+tmux send-keys -t "${WINDOW}" "cd $POCS" C-m
+tmux send-keys -t "${WINDOW}" "bin/peas_shell" C-m
+sleep 10s
+tmux send-keys -t "${WINDOW}" "display_config" C-m
+sleep 1s
+tmux send-keys -t "${WINDOW}" "load_environment" C-m
+sleep 5s
+tmux send-keys -t "${WINDOW}" "load_weather" C-m
+sleep 5s
+tmux send-keys -t "${WINDOW}" "start" C-m
+sleep 20s
+tmux send-keys -t "${WINDOW}" "status" C-m
+sleep 10s
+tmux send-keys -t "${WINDOW}" "last_reading environment" C-m
+sleep 10s
+tmux send-keys -t "${WINDOW}" "last_reading weather" C-m
+
+echo "Done at $(date)"

scripts/startup/start_pocs.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+WINDOW="${1}"
+echo "Running $(basename "${0}") at $(date), WINDOW=${WINDOW}"
+
+# Wait for bash to be ready (not necessary, but makes
+# the window look tidier when you attach later).
+sleep 1s
+
+tmux send-keys -t "${WINDOW}" "date" C-m
+tmux send-keys -t "${WINDOW}" "cd $POCS" C-m
+tmux send-keys -t "${WINDOW}" "bin/pocs_shell" C-m
+sleep 10s
+tmux send-keys -t "${WINDOW}" "setup_pocs" C-m
+sleep 20s
+tmux send-keys -t "${WINDOW}" "display_config" C-m
+sleep 1s
+tmux send-keys -t "${WINDOW}" "run_pocs" C-m
+
+echo "Done at $(date)"

scripts/startup/su_panoptes.sh

@@ -0,0 +1,16 @@
+#!/bin/sh -ex
+
+# Start the PANOPTES software running as user $PANUSER in a detached
+# tmux session; this enables an admin to attach to that session
+# later, see the output and take control of all the shells.
+
+# Put the date & time into the log (e.g. /tmp/su_panoptes.log).
+echo
+echo "Running ${0} at $(/bin/date)"
+echo
+
+# Execute tmux_launch.sh in a login shell for user $PANUSER.
+/bin/su --login --command \
+     "${POCS}/scripts/startup/tmux_launch.sh" "${PANUSER}"
+
+echo "Done at $(/bin/date)"

scripts/startup/tmux_launch.sh

@@ -0,0 +1,48 @@
+#!/bin/bash -ex
+
+# This script is designed to be run from /etc/rc.local (or similar)
+# as user $PANUSER with the PANOPTES environment variables set.
+
+echo "Running ${BASH_SOURCE[0]} at $(date)"
+
+# cd to the directory of this script, as that is where the
+# other start scripts are located.
+cd "$(dirname "${BASH_SOURCE[0]}")"
+
+echo "Setting up logging..."
+
+# Setup a directory into which we can log.
+export STARTUP_LOG_DIR_SLASH="${PANLOG}/per-run/startup/"
+mkdir -p "${STARTUP_LOG_DIR_SLASH}"
+
+NOW="$(date +%Y%m%d-%H%M%S-%Z)"
+LOG_FILE="${STARTUP_LOG_DIR_SLASH}$(basename "${BASH_SOURCE[0]}" .sh)-${NOW}.log"
+
+exec 2> "${LOG_FILE}"  # send stderr to a log file
+exec 1>&2              # send stdout to the same log file
+set -x
+
+# Record important debugging info into the log file.
+# These environment variables do NOT all have to be set.
+
+echo "Running ${BASH_SOURCE[0]} at $(date)"
+echo "Current dir: $(pwd)"
+echo "Current user: ${USER}"
+echo "Current path: ${PATH}"
+echo "PANUSER: ${PANUSER}"
+echo "PANDIR: ${PANDIR}"
+echo "PANLOG: ${PANLOG}"
+echo "POCS: ${POCS}"
+echo "PAWS: ${PAWS}"
+echo "PIAA: ${PIAA}"
+echo "MATPLOTLIBRC: ${MATPLOTLIBRC}"
+# $- expands to the current option flags as specified upon invocation, by the
+# set built-in command, or those set by the shell itself (such as the -i).
+echo '$-:' "$-"
+
+# Create a detached (-d) tmux session called panoptes (-s),
+# with a scrollback buffer of 5000 lines.
+tmux set-option -g history-limit 5000 \; new-session -d \
+                -s panoptes ./start_panoptes_in_tmux.sh
+
+exit