feat(compute,api,update-server): Move system configs out of Dockerfile (

#2073)

Overhaul the container configuration and initialization to remove as much as
possible from the Dockerfile and the volatile storage, preferring instead to
live in files that are included in the api server wheel and unpacked to /data
during initialization.

Most configuration and setup scripts are no longer COPYd into /usr/local/bin.
Instead, they are packed into the api server’s wheel (opentrons) in a
/resources subdirectory. This resources subdirectory is recursive-included
into the manifest, so putting a file into the directory will include it in the
wheel.

Because wheels do not run arbitrary code when they are installed, there is some
external tooling to get the files out of wheel/resources and into their eventual
home in /data/system.

* Provisioning

The compute/find_module_path.py is an executable python script that uses
importlib to find the location of the opentrons module as Python would do (so it
always gets the right one) without importing opentrons, which has significant
side effects.

The opentrons/api/opentrons/resources/scripts/provision-api-resources script
uses find_module_path to find the current opentrons module and copy everything
in /resources in the package to /data/system (the OT_CONFIG_DIR). This includes
all the system configurations and scripts that were previously in compute/.

The provision script is designed to be invoked during updates. This happens
during the first boot of the container (more on this later) and during a
software update via the update server.

In the case of the update-server driven update, because we assume a user is
never updating to 3.3 from 3.0 on a 3.0 Dockerfile, we can assume the provision
script is in /data/system/scripts and is therefore in the path. The update
server therefore invokes the provision script after installing a new api server,
the provision script finds the right (newly-installed) opentrons module, and
updates /data/resources with the new scripts.

In the case of the first container boot, we rely on the one piece of
initialization left in compute/. compute/container_setup.sh does the first-boot
check that setup.sh used to do, and in addition to removing old api server
installations and updating the cached container ID it runs the provision script
- this time from the module installed in /usr/local/lib.

* Container Initialization

The Dockerfile CMD has been slightly simplified by symlinking the environment
file into /etc/profile.d. This requires the use of bash’s -l flag to get a login
shell in the docker container.

Because the symlinks to the environment must be present when the shell starts,
the environment script (opentrons/api/opentrons/resources/ot-environ.sh) is
linked into /etc/profile.d by the Dockerfile (the rest of the configuration file
symlinks are in opentrons/api/opentrons/resources/scripts/setup.sh). It is also
linked twice - once from /data/system, which will be empty when the container
boots for the first time; and once from the system installation of the api
server in /usr/local/lib, as a fallback. The one-time environment variables
ensure it is only executed once.

Then, the container setup script runs. In most cases it doesn’t do much, but see
above for the behavior during the first container boot.

Since /data/system/scripts is in the path, we still call setup.sh and start.sh.
In addition to what it used to do, setup.sh now makes symlinks for most of the
configuration files we had been COPYing into Docker. These are now in
/data/system. start.sh is pretty similar to before.

* Container Building

The Dockerfile is now parameterized to make building a local container slightly
easier. Invoking docker build with no arguments builds a container for the
pi (it has to, since this is what Resin does). Invoking docker build with the
arguments to change the base image and clear RUNNING_ON_PI will build a
container for local machines. To make this easier, there is a new top-level
Makefile target, api-local-container, that invokes docker build with the correct
arguments. Note that the local container still doesn’t 100% work; it needs more
love to get over things like not having a dbus socket on all hosts.

* Misc

- Sweet new motd
- Removed some of the nmcli commands used for janitoring the static ipv6 routes
- Removed the nginx root server block and deleted the static files it was
- serving
- hardcoded some ports to get away from infinite env vars we can’t trust
- Fixed an issue in the update server where it was sending the repr() of
  tracebacks in 500 messages for /server/update. Now it sends the result of traceback.format_tb()

Closes #1114

Dockerfile

@@ -1,32 +1,15 @@
+ARG base_image=resin/raspberrypi3-alpine-python:3.6-slim-20180120
 # Use this for local development on intel machines
 # FROM resin/amd64-alpine-python:3.6-slim-20180123
 
+
 # Use this for running on a robot
-FROM resin/raspberrypi3-alpine-python:3.6-slim-20180120
-
-ENV RUNNING_ON_PI=1
-# This is used by D-Bus clients such as Network Manager cli, announce_mdns
-# connecting to Host OS services
-ENV DBUS_SYSTEM_BUS_ADDRESS=unix:path=/host/run/dbus/system_bus_socket
-# Add persisted data directory where new python packages are being installed
-ENV PYTHONPATH=$PYTHONPATH/data/packages/usr/local/lib/python3.6/site-packages
-ENV PATH=$PATH:/data/packages/usr/local/bin
-# Port name for connecting to smoothie over serial, i.e. /dev/ttyAMA0
-ENV OT_SMOOTHIE_ID=AMA
-ENV OT_SERVER_PORT=31950
-ENV OT_UPDATE_PORT=34000
-# File path to unix socket API server is listening
-ENV OT_SERVER_UNIX_SOCKET_PATH=/tmp/aiohttp.sock
-
-# Static IPv6 used on Ethernet interface for USB connectivity
-ENV ETHERNET_NETWORK_PREFIX=169.254
-ENV ETHERNET_NETWORK_PREFIX_LENGTH=16
+FROM $base_image
 
 # See compute/README.md for details. Make sure to keep them in sync
 RUN apk add --update \
       util-linux \
       vim \
-      radvd \
       dropbear \
       dropbear-scp \
       gnupg \
@@ -59,14 +42,14 @@ RUN pip install --force-reinstall \
 # Copy server files and data into the container. Note: any directories that
 # you wish to copy into the container must be excluded from the .dockerignore
 # file, or you will encounter a copy error
-ENV LABWARE_DEF /etc/labware
-ENV AUDIO_FILES /etc/audio
-ENV USER_DEFN_ROOT /data/user_storage/opentrons_data/labware
+
+COPY ./compute/container_setup.sh /usr/local/bin/container_setup.sh
+
 COPY ./shared-data/robot-data /etc/robot-data
-COPY ./compute/conf/jupyter_notebook_config.py /root/.jupyter/
 COPY ./shared-data/definitions /etc/labware
-COPY ./audio/ /etc/audio
 COPY ./api /tmp/api
+# Make our shared data available for the api setup.py
+COPY ./shared-data /tmp/shared-data
 COPY ./api-server-lib /tmp/api-server-lib
 COPY ./update-server /tmp/update-server
 COPY ./compute/avahi_tools /tmp/avahi_tools
@@ -81,6 +64,7 @@ RUN pipenv install /tmp/api-server-lib --system && \
     rm -rf /tmp/api && \
     rm -rf /tmp/api-server-lib && \
     rm -rf /tmp/update-server && \
+    rm -rf /tmp/shared-data && \
     rm -rf /tmp/avahi_tools
 
 # Redirect nginx logs to stdout and stderr
@@ -90,49 +74,12 @@ RUN ln -sf /dev/stdout /var/log/nginx/access.log && \
 # Use udev rules file from opentrons_data
 RUN ln -sf /data/user_storage/opentrons_data/95-opentrons-modules.rules /etc/udev/rules.d/95-opentrons-modules.rules
 
-# GPG public key to verify signed packages
-COPY ./compute/opentrons.asc .
-RUN gpg --import opentrons.asc && rm opentrons.asc
-
-# Everything you want in /usr/local/bin goes into compute/scripts
-COPY ./compute/scripts/* /usr/local/bin/
-
-# All configuration files live in compute/etc and dispatched here
-COPY ./compute/conf/inetd.conf /etc/
-COPY ./compute/conf/nginx.conf /etc/nginx/nginx.conf
-COPY ./compute/static /usr/share/nginx/html
-
 # Logo for login shell
 COPY ./compute/opentrons.motd /etc/motd
 
-# Replace placeholders with actual environment variable values
-RUN sed -i "s/{OT_SERVER_PORT}/$OT_SERVER_PORT/g" /etc/nginx/nginx.conf && \
-    sed -i "s#{OT_SERVER_UNIX_SOCKET_PATH}#$OT_SERVER_UNIX_SOCKET_PATH#g" /etc/nginx/nginx.conf
-
-# All newly installed packages will go to persistent storage
-ENV PIP_ROOT /data/packages
-
 # Generate keys for dropbear
-RUN ssh_key_gen.sh
-
-# Generate the id that we will later check to see if that's the
-# new container and that local Opentrons API package should be deleted
-# and persist all environment variables from the docker definition,
-# because they are sometimes not picked up from PID 1
-RUN echo "export CONTAINER_ID=$(uuidgen)" >> /etc/profile && \
-    echo "export OT_SETTINGS_DIR=$OT_SETTINGS_DIR" >> /etc/profile && \
-    echo "export OT_SERVER_PORT=$OT_SERVER_PORT" >> /etc/profile && \
-    echo "export OT_SERVER_UNIX_SOCKET_PATH=$OT_SERVER_UNIX_SOCKET_PATH" >> /etc/profile && \
-    echo "export PIP_ROOT=$PIP_ROOT" >> /etc/profile && \
-    echo "export LABWARE_DEF=$LABWARE_DEF" >> /etc/profile && \
-    echo "export USER_DEFN_ROOT=$USER_DEFN_ROOT" >> /etc/profile && \
-    echo "export AUDIO_FILES=$AUDIO_FILES" >> /etc/profile && \
-    echo "export PIPENV_VENV_IN_PROJECT=$PIPENV_VENV_IN_PROJECT" >> /etc/profile && \
-    echo "export DBUS_SYSTEM_BUS_ADDRESS=$DBUS_SYSTEM_BUS_ADDRESS" >> /etc/profile && \
-    echo "export PYTHONPATH=$PYTHONPATH" >> /etc/profile && \
-    echo "export PATH=$PATH" >> /etc/profile && \
-    echo "export RUNNING_ON_PI=$RUNNING_ON_PI" >> /etc/profile && \
-    echo "export OT_SMOOTHIE_ID=$OT_SMOOTHIE_ID" >> /etc/profile
+COPY ./compute/ssh_key_gen.sh /tmp/
+RUN /tmp/ssh_key_gen.sh
 
 # Updates, HTTPS (for future use), API, SSH for link-local over USB
 EXPOSE 80 443 31950
@@ -142,10 +89,28 @@ STOPSIGNAL SIGTERM
 # For backward compatibility, udev is enabled by default
 ENV UDEV on
 
+RUN echo "export CONTAINER_ID=$(uuidgen)" | tee -a /etc/profile.d/opentrons.sh
+
+# The one link we have to make in the dockerfile still to make sure we get our
+# environment variables
+COPY ./compute/find_python_module_path.py /usr/local/bin/
+RUN ln -sf /data/system/ot-environ.sh /etc/profile.d/00-persistent-ot-environ.sh &&\
+    ln -sf `find_python_module_path.py opentrons`/resources/ot-environ.sh /etc/profile.d/01-builtin-ot-environ.sh
+
+
+# This configuration is used both by both the build and runtime so it has to
+# be here. When building a container for local use, set this to 0. If set to
+# 0, ENABLE_VIRTUAL_SMOOTHIE will be set at runtime automatically
+ARG running_on_pi=1
+ENV RUNNING_ON_PI=$running_on_pi
+
+ARG data_mkdir_path_slash_if_none=/
+RUN mkdir -p $data_mkdir_path_slash_if_none
+
 # For interactive one-off use:
 #   docker run --name opentrons -it opentrons /bin/sh
 # or uncomment:
 # CMD ["python", "-c", "while True: pass"]
-CMD ["bash", "-c", "source /etc/profile && setup.sh && exec start.sh"]
+CMD ["bash", "-lc", "container_setup.sh && setup.sh && exec start.sh"]
 
 # Using Resin base image's default entrypoint and init system- tini

Makefile

@@ -58,6 +58,10 @@ push-api:
 	$(MAKE) -C $(API_DIR) push
 	$(MAKE) -C $(API_DIR) restart
 
+.PHONY: api-local-container
+api-local-container:
+	docker build --no-cache --build-arg base_image=resin/amd64-alpine-python:3.6-slim-20180123 --build-arg running_on_pi=0 --build-arg data_mkdir_path_slash_if_none=/data/system .
+
 # all tests
 .PHONY: test
 test: test-py test-js

api/MANIFEST.in

@@ -5,3 +5,4 @@ include opentrons/config/smoothie/config_one_pro_plus
 include opentrons/config/modules/avrdude.conf
 include opentrons/config/modules/95-opentrons-modules.rules
 include opentrons/config/pipette-config.json
+recursive-include opentrons/resources *

compute/conf/nginx.conf → api/opentrons/resources/nginx.conf

@@ -14,8 +14,8 @@ http {
     server {
         # Values in {} get replaced by environment variable during
         # container build
-        listen [::]:{OT_SERVER_PORT};
-        listen 0.0.0.0:{OT_SERVER_PORT};
+        listen [::]:31950;
+        listen 0.0.0.0:31950;
 
         client_body_in_file_only off;
         client_body_buffer_size  128k;
@@ -30,7 +30,7 @@ http {
             proxy_set_header         Upgrade $http_upgrade;
             proxy_set_header         Connection "upgrade";
             proxy_read_timeout       1h;
-            proxy_pass               http://unix:{OT_SERVER_UNIX_SOCKET_PATH};
+            proxy_pass               http://unix:/tmp/aiohttp.sock;
         }
 
         location /server {
@@ -41,15 +41,4 @@ http {
             proxy_pass               http://127.0.0.1:34000;
         }
     }
-
-    server {
-        root /usr/share/nginx/html;
-
-        client_max_body_size 100m;
-        listen [::] default_server ipv6only=on;
-
-        location / {
-            index index.htm index.html;
-        }
-    }
 }

api/opentrons/resources/ot-environ.sh

@@ -0,0 +1,33 @@
+# Set up the environment for the OT2.
+# This is sourced by the system login shell profile by symlinks placed in
+# /etc/profile.d by the Dockerfile.
+
+if [ -z $OT_ENVIRON_SET_UP ]; then
+    echo "[ $0 ] Configuring environment"
+
+    # Make sure pip installs things into /data
+    export PIP_ROOT=/data/packages
+
+    export OT_CONFIG_PATH=/data/system
+
+    # Required for proper pipenv operation
+    export PIPENV_VENV_IN_PROJECT=true
+    # This is used by D-Bus clients such as Network Manager cli, announce_mdns
+    # connecting to Host OS services
+    export DBUS_SYSTEM_BUS_ADDRESS=unix:path=/host/run/dbus/system_bus_socket
+    export PYTHONPATH=$PYTHONPATH:/data/packages/usr/local/lib/python3.6/site-packages
+    export PATH=$PATH:/data/packages/usr/local/bin:$OT_CONFIG_PATH/scripts
+
+    # TODO(seth, 8/15/2018): These are almost certainly unused and should be hardcoded
+    # if they are in fact still used
+    export OT_SETTINGS_DIR=""
+    export OT_SERVER_UNIX_SOCKET_PATH=/tmp/aiohttp.sock
+    export LABWARE_DEF=/etc/labware
+    export AUDIO_FILES=/etc/audio
+    export USER_DEFN_ROOT=/data/user_storage/opentrons_data/labware
+    export OT_SMOOTHIE_ID=AMA
+    export OT_ENVIRON_SET_UP=1
+    echo "[ $0 ] Environment configuration done"
+else
+    echo "[ $0 ] Environment already configured"
+fi

api/opentrons/resources/scripts/provision-api-resources

@@ -0,0 +1,31 @@
+#!/usr/bin/env python
+""" Copy everything here (except this script) into /data/system.
+
+This should be run
+- On the first boot of a new container (handled by `container_setup.sh`)
+- When a new version of the API server is installed by runapp (handled by `setup.py`) in the API server wheel
+"""
+
+import os
+import shutil
+import sys
+
+sys.path.append('/usr/local/bin')
+import find_python_module_path
+
+def provision():
+    """ Should be called the first time a given version of the server is run in a container.
+
+    Should not be called if the server is not running in a container.
+    """
+    provision_from_module = find_python_module_path.find_module('opentrons')
+    provision_from_resources = os.path.join(provision_from_module, 'resources')
+    print("Provisioning config and initialization from {}"
+          .format(provision_from_resources))
+    config_dir = os.environ.get('OT_CONFIG_PATH', '/data/system')
+    if os.path.exists(config_dir):
+        shutil.rmtree(config_dir)
+    shutil.copytree(provision_from_resources, config_dir)
+
+if __name__ == '__main__':
+    provision()

api/opentrons/resources/scripts/setup.sh

@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+
+echo "[ $0 ] API server setup beginning"
+
+if [ ! -z $RUNNING_ON_PI ] ; then
+    echo "[ $0 ] Container running on raspi detected, running system setup"
+    mount_usb.py
+    setup_gpio.py
+
+    # Cleanup any connections. This will leave only wlan0
+    nmcli --terse --fields uuid,device connection show | sed -rn "s/(.*):(--)/\1/p" | xargs nmcli connection del || true
+    nmcli --terse --fields uuid,device connection show | sed -rn "s/(.*):(eth0)/\1/p" | xargs nmcli connection del || true
+
+
+    # nmcli makes an async call which might not finish before next network-related
+    # operation starts. There is no graceful way to await for D-BUS event in shell
+    # hence sleep is added to avoid race condition
+    sleep 1
+    nmcli con add con-name "static-eth0" ifname eth0 type ethernet ipv4.method link-local
+else
+    echo "[ $0 ] Container running locally"
+fi
+
+echo "[$0 ] Creating config file links (OT_CONFIG_PATH=$OT_CONFIG_PATH )..."
+
+ln -sf $OT_CONFIG_PATH/jupyter /root/.jupyter
+ln -sf $OT_CONFIG_PATH/audio /etc/audio
+rm /etc/nginx/nginx.conf
+ln -sf $OT_CONFIG_PATH/nginx.conf /etc/nginx/nginx.conf
+ln -sf $OT_CONFIG_PATH/inetd.conf /etc/inetd.conf
+mkdir -p /run/nginx
+
+echo "[ $0 ] API server setup done"

compute/scripts/setup_gpio.py → api/opentrons/resources/scripts/setup_gpio.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python
 from time import sleep
 
 from opentrons.drivers.rpi_drivers import gpio

compute/scripts/start.sh → api/opentrons/resources/scripts/start.sh

@@ -1,36 +1,42 @@
 #!/usr/bin/env bash
-
+echo "[ $0 ] API server starting"
 # mdns announcement
-announce_mdns.py &
+if [ ! -z $RUNNING_ON_PI ]; then
+    echo "[ $0 ] MDNS beginning"
+    announce_mdns.py &
+fi
 
+echo "[ $0 ] Starting nginx"
 # serve static pages and proxy HTTP services
 nginx
 
+echo "[ $0 ] Starting inetd"
 # enable SSH over ethernet
 inetd -e /etc/inetd.conf
 
+echo "[ $0 ] Running user boot scripts"
 # If user boot script exists, run it
 mkdir -p /data/boot.d
 run-parts /data/boot.d
 
-echo "Starting Opentrons update server"
-python -m otupdate --debug --port $OT_UPDATE_PORT &
+echo "[ $0 ] Starting Opentrons update server"
+python -m otupdate --debug --port 34000 &
 
-echo "Starting Jupyter Notebook server"
+echo "[ $0 ] Starting Jupyter Notebook server"
 mkdir -p /data/user_storage/opentrons_data/jupyter
 jupyter notebook --allow-root &
 
 # Check if config exists, and alert if not found
-echo "Checking for deck calibration data..."
+echo "[ $0 ] Checking for deck calibration data..."
 config_path=`python -c "from opentrons import config; print(config.get_config_index().get('deckCalibrationFile'))"`
 
 if [ ! -e "$config_path" ]; then
     echo $config_path
-    echo "Config file not found. Please perform factory calibration and then restart robot"
+    echo "[ $0 ] Config file not found. Please perform factory calibration and then restart robot"
 fi
 
 export ENABLE_NETWORKING_ENDPOINTS=true
-echo "Starting Opentrons API server"
+echo "[ $0 ] Starting Opentrons API server"
 python -m opentrons.server.main -U $OT_SERVER_UNIX_SOCKET_PATH opentrons.server.main:init
-echo "Server exited unexpectedly. Please power-cycle the machine, and contact Opentrons support."
+echo "[ $0 ] Server exited unexpectedly. Please power-cycle the machine, and contact Opentrons support."
 while true; do sleep 1; done

api/pylama.ini

@@ -26,5 +26,8 @@ skip = 1
 [pylama:opentrons\_version.py]
 skip = 1
 
+[pylama:opentrons/resources/jupyter/jupyter_notebook_config.py]
+skip = 1
+